[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:24:13 dist-test-slave-47qt krb5kdc[8357](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:24:13 dist-test-slave-47qt krb5kdc[8357](info): set up 2 sockets
Jun 18 23:24:13 dist-test-slave-47qt krb5kdc[8357](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260618 23:24:14.595719 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:24:14 dist-test-slave-47qt krb5kdc[8357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:24:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:24:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.39.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
I20260618 23:24:14.715929 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:35535
--webserver_interface=127.8.39.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:44947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:35535
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:14.795553 8370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:14.795780 8370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:14.795814 8370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:14.796985 8370 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:24:14.797053 8370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:14.797077 8370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:14.797106 8370 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:24:14.797125 8370 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:24:14.798616 8370 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:44947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:35535
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:35535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:14.798874 8370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:14.799096 8370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:14.804469 8376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:14.804494 8375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:14.804520 8370 server_base.cc:1034] running on GCE node
W20260618 23:24:14.804494 8378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:14.805222 8370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:14.805512 8370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:14.806712 8370 hybrid_clock.cc:648] HybridClock initialized: now 1781825054806678 us; error 39 us; skew 500 ppm
Jun 18 23:24:14 dist-test-slave-47qt krb5kdc[8357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825054, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:14.809950 8370 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:24:14.810474 8370 webserver.cc:460] Webserver started at http://127.8.39.190:39957/ using document root <none> and password file <none>
I20260618 23:24:14.810720 8370 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:14.810791 8370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:14.810942 8370 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:14.811997 8370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "4a32d4d40b6a403b9a6d97076ec7eac8"
format_stamp: "Formatted at 2026-06-18 23:24:14 on dist-test-slave-47qt"
server_key: "9659013197b6a8c19f3e157ad7c57f1e"
server_key_iv: "2061abbd26eb82df27eb3d9c4f7caade"
server_key_version: "encryptionkey@0"
I20260618 23:24:14.812361 8370 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "4a32d4d40b6a403b9a6d97076ec7eac8"
format_stamp: "Formatted at 2026-06-18 23:24:14 on dist-test-slave-47qt"
server_key: "9659013197b6a8c19f3e157ad7c57f1e"
server_key_iv: "2061abbd26eb82df27eb3d9c4f7caade"
server_key_version: "encryptionkey@0"
I20260618 23:24:14.813817 8370 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:24:14.814781 8385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:14.815011 8370 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:14.815137 8370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "4a32d4d40b6a403b9a6d97076ec7eac8"
format_stamp: "Formatted at 2026-06-18 23:24:14 on dist-test-slave-47qt"
server_key: "9659013197b6a8c19f3e157ad7c57f1e"
server_key_iv: "2061abbd26eb82df27eb3d9c4f7caade"
server_key_version: "encryptionkey@0"
I20260618 23:24:14.815214 8370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:14.831319 8370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:14.833317 8370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:14.833470 8370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:14.837644 8370 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:35535
I20260618 23:24:14.837684 8437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:35535 every 8 connection(s)
I20260618 23:24:14.838037 8370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:24:14.838857 8438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:14.840793 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 8370
I20260618 23:24:14.840906 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:24:14.841284 8438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a32d4d40b6a403b9a6d97076ec7eac8: Bootstrap starting.
I20260618 23:24:14.841714 8350 external_mini_cluster.cc:1428] Setting key bc732b1bbd9c82ebb5143f50fdef5534
I20260618 23:24:14.841971 8438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a32d4d40b6a403b9a6d97076ec7eac8: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:14.842276 8438 log.cc:824] T 00000000000000000000000000000000 P 4a32d4d40b6a403b9a6d97076ec7eac8: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:14.843050 8438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a32d4d40b6a403b9a6d97076ec7eac8: No bootstrap required, opened a new log
I20260618 23:24:14.844569 8438 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a32d4d40b6a403b9a6d97076ec7eac8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a32d4d40b6a403b9a6d97076ec7eac8" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 35535 } }
I20260618 23:24:14.844729 8438 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a32d4d40b6a403b9a6d97076ec7eac8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:14.844774 8438 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a32d4d40b6a403b9a6d97076ec7eac8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a32d4d40b6a403b9a6d97076ec7eac8, State: Initialized, Role: FOLLOWER
I20260618 23:24:14.844907 8438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a32d4d40b6a403b9a6d97076ec7eac8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a32d4d40b6a403b9a6d97076ec7eac8" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 35535 } }
I20260618 23:24:14.844969 8438 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a32d4d40b6a403b9a6d97076ec7eac8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:24:14.845021 8438 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a32d4d40b6a403b9a6d97076ec7eac8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:24:14.845062 8438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a32d4d40b6a403b9a6d97076ec7eac8 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:14.846076 8438 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a32d4d40b6a403b9a6d97076ec7eac8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a32d4d40b6a403b9a6d97076ec7eac8" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 35535 } }
I20260618 23:24:14.846196 8438 leader_election.cc:302] T 00000000000000000000000000000000 P 4a32d4d40b6a403b9a6d97076ec7eac8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a32d4d40b6a403b9a6d97076ec7eac8; no voters:
I20260618 23:24:14.846339 8438 leader_election.cc:288] T 00000000000000000000000000000000 P 4a32d4d40b6a403b9a6d97076ec7eac8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:24:14.846387 8443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a32d4d40b6a403b9a6d97076ec7eac8 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:14.846520 8443 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a32d4d40b6a403b9a6d97076ec7eac8 [term 1 LEADER]: Becoming Leader. State: Replica: 4a32d4d40b6a403b9a6d97076ec7eac8, State: Running, Role: LEADER
I20260618 23:24:14.846573 8438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a32d4d40b6a403b9a6d97076ec7eac8 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:24:14.846603 8443 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a32d4d40b6a403b9a6d97076ec7eac8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a32d4d40b6a403b9a6d97076ec7eac8" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 35535 } }
I20260618 23:24:14.846894 8444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a32d4d40b6a403b9a6d97076ec7eac8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a32d4d40b6a403b9a6d97076ec7eac8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a32d4d40b6a403b9a6d97076ec7eac8" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 35535 } } }
I20260618 23:24:14.846966 8444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a32d4d40b6a403b9a6d97076ec7eac8 [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:14.847187 8445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a32d4d40b6a403b9a6d97076ec7eac8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a32d4d40b6a403b9a6d97076ec7eac8. Latest consensus state: current_term: 1 leader_uuid: "4a32d4d40b6a403b9a6d97076ec7eac8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a32d4d40b6a403b9a6d97076ec7eac8" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 35535 } } }
I20260618 23:24:14.847524 8445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a32d4d40b6a403b9a6d97076ec7eac8 [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:14.847512 8448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:24:14.848029 8448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:24:14.849524 8448 catalog_manager.cc:1347] Generated new cluster ID: 03c83836af90448d865987e3282abe8e
I20260618 23:24:14.849577 8448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 18 23:24:14 dist-test-slave-47qt krb5kdc[8357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:14.859277 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:14.842720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44948 (local address 127.8.39.190:35535)
0618 23:24:14.842895 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:14.842904 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:14.843214 (+ 310us) server_negotiation.cc:407] Connection header received
0618 23:24:14.844846 (+ 1632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:14.844852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:14.844905 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:14.845033 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:14.847616 (+ 2583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:14.849389 (+ 1773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:14.850814 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:14.851321 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:14.851580 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:14.855118 (+ 3538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:14.855122 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:14.855122 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:14.855141 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:14.856972 (+ 1831us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:14.857560 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:14.857563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:14.857564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:14.857687 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:14.857994 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:14.857995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:14.857995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:14.858362 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:14.858526 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:14.858913 (+ 387us) server_negotiation.cc:299] Negotiation successful
0618 23:24:14.859062 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260618 23:24:14.862758 8448 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:24:14.863530 8448 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:24:14.870877 8448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a32d4d40b6a403b9a6d97076ec7eac8: Generated new TSK 0
I20260618 23:24:14.871063 8448 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.39.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
I20260618 23:24:14.901752 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.129:0
--local_ip_for_outbound_sockets=127.8.39.129
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:35535
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:44947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:15.002802 8464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:15.003003 8464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:15.003027 8464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:15.004426 8464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:15.004489 8464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:15.004519 8464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.129
I20260618 23:24:15.006100 8464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:44947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:35535
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.129
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:15.006368 8464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:15.006659 8464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:15.012985 8469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:15.013093 8464 server_base.cc:1034] running on GCE node
W20260618 23:24:15.012969 8470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:15.012965 8472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:15.013438 8464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:15.013656 8464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:15.014856 8464 hybrid_clock.cc:648] HybridClock initialized: now 1781825055014840 us; error 44 us; skew 500 ppm
Jun 18 23:24:15 dist-test-slave-47qt krb5kdc[8357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825055, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:15.017939 8464 init.cc:377] Logged in from keytab as kudu/127.8.39.129@KRBTEST.COM (short username kudu)
I20260618 23:24:15.018424 8464 webserver.cc:460] Webserver started at http://127.8.39.129:38445/ using document root <none> and password file <none>
I20260618 23:24:15.018673 8464 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:15.018741 8464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:15.018882 8464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:15.019934 8464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-0/data/instance:
uuid: "72ec74e11c624fb8975b018aa4d61a9a"
format_stamp: "Formatted at 2026-06-18 23:24:15 on dist-test-slave-47qt"
server_key: "3cad0cd26db641202801165f5ece9833"
server_key_iv: "ce5315e2291fabfe8202a0bdd0c487fb"
server_key_version: "encryptionkey@0"
I20260618 23:24:15.020298 8464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance:
uuid: "72ec74e11c624fb8975b018aa4d61a9a"
format_stamp: "Formatted at 2026-06-18 23:24:15 on dist-test-slave-47qt"
server_key: "3cad0cd26db641202801165f5ece9833"
server_key_iv: "ce5315e2291fabfe8202a0bdd0c487fb"
server_key_version: "encryptionkey@0"
I20260618 23:24:15.021751 8464 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:24:15.022624 8479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:15.023008 8464 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:15.023222 8464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-0/wal
uuid: "72ec74e11c624fb8975b018aa4d61a9a"
format_stamp: "Formatted at 2026-06-18 23:24:15 on dist-test-slave-47qt"
server_key: "3cad0cd26db641202801165f5ece9833"
server_key_iv: "ce5315e2291fabfe8202a0bdd0c487fb"
server_key_version: "encryptionkey@0"
I20260618 23:24:15.023342 8464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:15.043031 8464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:15.045531 8464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:15.045704 8464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:15.045954 8464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:15.046252 8464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:15.046306 8464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:15.046365 8464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:15.046396 8464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:15.054404 8464 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.129:40519
I20260618 23:24:15.054535 8592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.129:40519 every 8 connection(s)
I20260618 23:24:15.054814 8464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
I20260618 23:24:15.057469 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 8464
I20260618 23:24:15.057585 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance
I20260618 23:24:15.057744 8350 external_mini_cluster.cc:1428] Setting key 168726f8479c6b0a022b3c7574e4b219
Jun 18 23:24:15 dist-test-slave-47qt krb5kdc[8357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825055, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.39.130@KRBTEST.COM; defaulting to no policy
I20260618 23:24:15.067000 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:15.055762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:40135 (local address 127.8.39.190:35535)
0618 23:24:15.055827 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:15.055829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:15.055999 (+ 170us) server_negotiation.cc:407] Connection header received
0618 23:24:15.056545 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:15.056546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:15.056560 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:15.056615 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:15.057722 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:15.058461 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:15.060327 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:15.060672 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:15.060759 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:15.063271 (+ 2512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:15.063273 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:15.063273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:15.063281 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:15.064956 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:15.065585 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:15.065586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:15.065586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:15.065649 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:15.065995 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:15.065997 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:15.065997 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:15.066285 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:15.066375 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:15.066766 (+ 391us) server_negotiation.cc:299] Negotiation successful
0618 23:24:15.066873 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260618 23:24:15.067384 8595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:15.055741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:35535 (local address 127.8.39.129:40135)
0618 23:24:15.055897 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:15.055932 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:15.056419 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:15.056681 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:15.056685 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:15.057379 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:15.057575 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:15.057580 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:15.058649 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:15.058651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:15.060220 (+ 1569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:15.060225 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:15.060774 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:15.060776 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:15.060877 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:15.061570 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:15.061581 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:15.063170 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:15.065182 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:15.065189 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:15.065192 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:15.065439 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:15.065727 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:15.065728 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:15.065728 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:15.065873 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:15.066366 (+ 493us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:15.066374 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:15.066575 (+ 201us) client_negotiation.cc:769] Sending connection context
0618 23:24:15.066708 (+ 133us) client_negotiation.cc:240] Negotiation successful
0618 23:24:15.067023 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260618 23:24:15.068043 8593 heartbeater.cc:347] Connected to a master server at 127.8.39.190:35535
I20260618 23:24:15.068154 8593 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:15.068297 8593 heartbeater.cc:511] Master 127.8.39.190:35535 requested a full tablet report, sending...
I20260618 23:24:15.069173 8402 ts_manager.cc:194] Registered new tserver with Master: 72ec74e11c624fb8975b018aa4d61a9a (127.8.39.129:40519)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.130@KRBTEST.COM" created.
I20260618 23:24:15.069912 8402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.129@KRBTEST.COM'} at 127.8.39.129:40135
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
I20260618 23:24:15.087592 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.130:0
--local_ip_for_outbound_sockets=127.8.39.130
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:35535
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:44947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:15.166873 8598 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:15.167088 8598 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:15.167110 8598 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:15.168176 8598 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:15.168215 8598 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:15.168241 8598 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.130
I20260618 23:24:15.169751 8598 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:44947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:35535
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8598
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.130
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:15.169952 8598 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:15.170114 8598 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:15.175092 8603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:15.175069 8604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:15.175312 8598 server_base.cc:1034] running on GCE node
W20260618 23:24:15.175092 8606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:15.175655 8598 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:15.175896 8598 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:15.177203 8598 hybrid_clock.cc:648] HybridClock initialized: now 1781825055177167 us; error 38 us; skew 500 ppm
Jun 18 23:24:15 dist-test-slave-47qt krb5kdc[8357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825055, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:15.180176 8598 init.cc:377] Logged in from keytab as kudu/127.8.39.130@KRBTEST.COM (short username kudu)
I20260618 23:24:15.180594 8598 webserver.cc:460] Webserver started at http://127.8.39.130:35075/ using document root <none> and password file <none>
I20260618 23:24:15.180845 8598 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:15.180912 8598 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:15.181059 8598 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:15.182106 8598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-1/data/instance:
uuid: "701ca44586734247bc52048e874a07ed"
format_stamp: "Formatted at 2026-06-18 23:24:15 on dist-test-slave-47qt"
server_key: "01580cebdc2d168a02a1270967db2699"
server_key_iv: "b641ed530c82e70e0f4e8088f4695f75"
server_key_version: "encryptionkey@0"
I20260618 23:24:15.182472 8598 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance:
uuid: "701ca44586734247bc52048e874a07ed"
format_stamp: "Formatted at 2026-06-18 23:24:15 on dist-test-slave-47qt"
server_key: "01580cebdc2d168a02a1270967db2699"
server_key_iv: "b641ed530c82e70e0f4e8088f4695f75"
server_key_version: "encryptionkey@0"
I20260618 23:24:15.183832 8598 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:24:15.184666 8613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:15.184850 8598 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:15.184952 8598 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-1/wal
uuid: "701ca44586734247bc52048e874a07ed"
format_stamp: "Formatted at 2026-06-18 23:24:15 on dist-test-slave-47qt"
server_key: "01580cebdc2d168a02a1270967db2699"
server_key_iv: "b641ed530c82e70e0f4e8088f4695f75"
server_key_version: "encryptionkey@0"
I20260618 23:24:15.185041 8598 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:15.194717 8598 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:15.196703 8598 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:15.196874 8598 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:15.197161 8598 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:15.197499 8598 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:15.197553 8598 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:15.197597 8598 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:15.197624 8598 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:15.203727 8598 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.130:38119
I20260618 23:24:15.203800 8726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.130:38119 every 8 connection(s)
I20260618 23:24:15.204205 8598 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
Jun 18 23:24:15 dist-test-slave-47qt krb5kdc[8357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825055, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:15.212477 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 8598
I20260618 23:24:15.212550 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance
I20260618 23:24:15.212824 8350 external_mini_cluster.cc:1428] Setting key 2b7226c1f6073ca0288b0d234df10cb3
I20260618 23:24:15.214648 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:15.204944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:47851 (local address 127.8.39.190:35535)
0618 23:24:15.205021 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:15.205023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:15.205353 (+ 330us) server_negotiation.cc:407] Connection header received
0618 23:24:15.205913 (+ 560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:15.205914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:15.205927 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:15.206019 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:15.207249 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:15.207795 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:15.208942 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:15.209337 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:15.209442 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:15.211570 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:15.211572 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:15.211572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:15.211579 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:15.213211 (+ 1632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:15.213667 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:15.213668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:15.213669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:15.213726 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:15.213994 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:15.213995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:15.213996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:15.214116 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:15.214192 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:15.214462 (+ 270us) server_negotiation.cc:299] Negotiation successful
0618 23:24:15.214552 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260618 23:24:15.214917 8729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:15.204951 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:35535 (local address 127.8.39.130:47851)
0618 23:24:15.205209 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:15.205257 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:15.205798 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:15.206097 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:15.206101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:15.206724 (+ 623us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:15.206921 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:15.206926 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:15.207900 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:15.207902 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:15.208827 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:15.208829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:15.209450 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:15.209451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:15.209598 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:15.210119 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:15.210127 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:15.211463 (+ 1336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:15.213304 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:15.213311 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:15.213313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:15.213608 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:15.213804 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:15.213805 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:15.213805 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:15.213919 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:15.214184 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:15.214190 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:15.214329 (+ 139us) client_negotiation.cc:769] Sending connection context
0618 23:24:15.214441 (+ 112us) client_negotiation.cc:240] Negotiation successful
0618 23:24:15.214613 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":91,"threads_started":1}
I20260618 23:24:15.215474 8727 heartbeater.cc:347] Connected to a master server at 127.8.39.190:35535
I20260618 23:24:15.215590 8727 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:15.215759 8727 heartbeater.cc:511] Master 127.8.39.190:35535 requested a full tablet report, sending...
I20260618 23:24:15.216161 8402 ts_manager.cc:194] Registered new tserver with Master: 701ca44586734247bc52048e874a07ed (127.8.39.130:38119)
I20260618 23:24:15.216706 8402 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.130@KRBTEST.COM'} at 127.8.39.130:47851
WARNING: no policy specified for kudu/127.8.39.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
I20260618 23:24:15.239279 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.131:0
--local_ip_for_outbound_sockets=127.8.39.131
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:35535
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:44947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:15.318862 8732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:15.319077 8732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:15.319101 8732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:15.320214 8732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:15.320251 8732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:15.320278 8732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.131
I20260618 23:24:15.322098 8732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:44947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:35535
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.131
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:15.322300 8732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:15.322471 8732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:15.327816 8737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:15.327855 8740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:15.327814 8738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:15.328230 8732 server_base.cc:1034] running on GCE node
I20260618 23:24:15.328632 8732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:15.329051 8732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:15.330260 8732 hybrid_clock.cc:648] HybridClock initialized: now 1781825055330216 us; error 40 us; skew 500 ppm
Jun 18 23:24:15 dist-test-slave-47qt krb5kdc[8357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825055, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:15.332942 8732 init.cc:377] Logged in from keytab as kudu/127.8.39.131@KRBTEST.COM (short username kudu)
I20260618 23:24:15.333498 8732 webserver.cc:460] Webserver started at http://127.8.39.131:36255/ using document root <none> and password file <none>
I20260618 23:24:15.333725 8732 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:15.333770 8732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:15.333860 8732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:15.334930 8732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-2/data/instance:
uuid: "a3e035e5d6cf44738bd30ce9bf66174c"
format_stamp: "Formatted at 2026-06-18 23:24:15 on dist-test-slave-47qt"
server_key: "6bc52011f9a0db9895ab68b32fa06604"
server_key_iv: "d90eabe24f473a064957e33ee44c353a"
server_key_version: "encryptionkey@0"
I20260618 23:24:15.335247 8732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance:
uuid: "a3e035e5d6cf44738bd30ce9bf66174c"
format_stamp: "Formatted at 2026-06-18 23:24:15 on dist-test-slave-47qt"
server_key: "6bc52011f9a0db9895ab68b32fa06604"
server_key_iv: "d90eabe24f473a064957e33ee44c353a"
server_key_version: "encryptionkey@0"
I20260618 23:24:15.336527 8732 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:24:15.337380 8747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:15.337595 8732 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:24:15.337672 8732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-2/wal
uuid: "a3e035e5d6cf44738bd30ce9bf66174c"
format_stamp: "Formatted at 2026-06-18 23:24:15 on dist-test-slave-47qt"
server_key: "6bc52011f9a0db9895ab68b32fa06604"
server_key_iv: "d90eabe24f473a064957e33ee44c353a"
server_key_version: "encryptionkey@0"
I20260618 23:24:15.337728 8732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:15.345011 8732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:15.347410 8732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:15.347524 8732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:15.347822 8732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:15.348130 8732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:15.348166 8732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:15.348207 8732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:15.348229 8732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:15.354686 8732 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.131:35197
I20260618 23:24:15.354735 8860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.131:35197 every 8 connection(s)
I20260618 23:24:15.355038 8732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
Jun 18 23:24:15 dist-test-slave-47qt krb5kdc[8357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825055, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:15.363642 8441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:15.355754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:34349 (local address 127.8.39.190:35535)
0618 23:24:15.355796 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:15.355798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:15.355976 (+ 178us) server_negotiation.cc:407] Connection header received
0618 23:24:15.356413 (+ 437us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:15.356413 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:15.356423 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:15.356485 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:15.357404 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:15.357883 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:15.358800 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:15.358995 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:15.359052 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:15.361166 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:15.361167 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:15.361167 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:15.361173 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:15.362397 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:15.362754 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:15.362755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:15.362771 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:15.362824 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:15.363099 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:15.363100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:15.363100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:15.363206 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:15.363265 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:15.363518 (+ 253us) server_negotiation.cc:299] Negotiation successful
0618 23:24:15.363579 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260618 23:24:15.363857 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:15.355684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:35535 (local address 127.8.39.131:34349)
0618 23:24:15.355843 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:15.355882 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:15.356338 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:15.356535 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:15.356538 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:15.357078 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:15.357323 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:15.357327 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:15.357970 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:15.357971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:15.358724 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:15.358725 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:15.359068 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:15.359069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:15.359168 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:15.359683 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:15.359691 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:15.361053 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:15.362475 (+ 1422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:15.362480 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:15.362481 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:15.362686 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:15.362891 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:15.362892 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:15.362913 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:15.363021 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:15.363269 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:15.363272 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:15.363373 (+ 101us) client_negotiation.cc:769] Sending connection context
0618 23:24:15.363476 (+ 103us) client_negotiation.cc:240] Negotiation successful
0618 23:24:15.363590 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260618 23:24:15.363960 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 8732
I20260618 23:24:15.364043 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance
I20260618 23:24:15.364228 8350 external_mini_cluster.cc:1428] Setting key 41ef0a3bd38af1b2bf814299058a4c2e
I20260618 23:24:15.364439 8861 heartbeater.cc:347] Connected to a master server at 127.8.39.190:35535
I20260618 23:24:15.364537 8861 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:15.364711 8861 heartbeater.cc:511] Master 127.8.39.190:35535 requested a full tablet report, sending...
I20260618 23:24:15.365062 8401 ts_manager.cc:194] Registered new tserver with Master: a3e035e5d6cf44738bd30ce9bf66174c (127.8.39.131:35197)
I20260618 23:24:15.365496 8401 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.131@KRBTEST.COM'} at 127.8.39.131:34349
I20260618 23:24:15.376263 8350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 18 23:24:15 dist-test-slave-47qt krb5kdc[8357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825055, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 18 23:24:15 dist-test-slave-47qt krb5kdc[8357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825055, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.39.129@KRBTEST.COM
I20260618 23:24:15.399756 8868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:15.391453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36600 (local address 127.8.39.129:40519)
0618 23:24:15.391628 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:15.391633 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:15.391643 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:24:15.391669 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:15.391671 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:15.391684 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:15.391750 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:15.392567 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:15.393404 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:15.394171 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:15.394491 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:15.394547 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:15.396887 (+ 2340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:15.396896 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:15.396896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:15.396913 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:15.398549 (+ 1636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:15.398932 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:15.398935 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:15.398936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:15.398996 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:15.399199 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:15.399200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:15.399201 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:15.399338 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:15.399513 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:15.399557 (+ 44us) server_negotiation.cc:299] Negotiation successful
0618 23:24:15.399633 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":88,"threads_started":1}
W20260618 23:24:15.400146 8507 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36600
Jun 18 23:24:15 dist-test-slave-47qt krb5kdc[8357](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 18 23:24:15 dist-test-slave-47qt krb5kdc[8357](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.129@KRBTEST.COM
I20260618 23:24:15.423691 8868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:15.416254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36610 (local address 127.8.39.129:40519)
0618 23:24:15.416323 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:15.416325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:15.416383 (+ 58us) server_negotiation.cc:407] Connection header received
0618 23:24:15.416440 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:15.416441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:15.416449 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:15.416502 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:15.417064 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:15.417717 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:15.418480 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:15.418692 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:15.418751 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:15.420951 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:15.420952 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:15.420952 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:15.420959 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:15.422444 (+ 1485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:15.422907 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:15.422909 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:15.422909 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:15.422962 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:15.423217 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:15.423218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:15.423218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:15.423313 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:15.423415 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:15.423561 (+ 146us) server_negotiation.cc:299] Negotiation successful
0618 23:24:15.423627 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260618 23:24:15.424238 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 8464
I20260618 23:24:15.429440 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 8598
I20260618 23:24:15.433676 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 8732
I20260618 23:24:15.437790 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 8370
2026-06-18T23:24:15Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1903 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:24:15 dist-test-slave-47qt krb5kdc[8875](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:24:15 dist-test-slave-47qt krb5kdc[8875](info): set up 2 sockets
Jun 18 23:24:15 dist-test-slave-47qt krb5kdc[8875](info): commencing operation
krb5kdc: starting...
W20260618 23:24:16.489948 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:24:16 dist-test-slave-47qt krb5kdc[8875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:24:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:24:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.39.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
I20260618 23:24:16.609813 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:38649
--webserver_interface=127.8.39.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:38649
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:16.690937 8888 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:16.691131 8888 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:16.691154 8888 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:16.692214 8888 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:24:16.692247 8888 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:16.692261 8888 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:16.692274 8888 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:24:16.692286 8888 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:24:16.693729 8888 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:38649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:38649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8888
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:16.693928 8888 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:16.694087 8888 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:16.699417 8893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:16.699407 8894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:16.699466 8888 server_base.cc:1034] running on GCE node
W20260618 23:24:16.699407 8896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:16.699797 8888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:16.700152 8888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:16.701324 8888 hybrid_clock.cc:648] HybridClock initialized: now 1781825056701305 us; error 50 us; skew 500 ppm
Jun 18 23:24:16 dist-test-slave-47qt krb5kdc[8875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825056, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:16.703850 8888 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:24:16.704348 8888 webserver.cc:460] Webserver started at http://127.8.39.190:35147/ using document root <none> and password file <none>
I20260618 23:24:16.704610 8888 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:16.704679 8888 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:16.704908 8888 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:16.706023 8888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "9b07bac5ee644116a5772d50a74338b2"
format_stamp: "Formatted at 2026-06-18 23:24:16 on dist-test-slave-47qt"
server_key: "f9c401f602d5201dbf99fb660ea6a41a"
server_key_iv: "a9cb1cf381dbadf0caf9e1e13d250273"
server_key_version: "encryptionkey@0"
I20260618 23:24:16.706377 8888 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "9b07bac5ee644116a5772d50a74338b2"
format_stamp: "Formatted at 2026-06-18 23:24:16 on dist-test-slave-47qt"
server_key: "f9c401f602d5201dbf99fb660ea6a41a"
server_key_iv: "a9cb1cf381dbadf0caf9e1e13d250273"
server_key_version: "encryptionkey@0"
I20260618 23:24:16.707865 8888 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:24:16.708636 8903 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:16.708842 8888 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:16.708958 8888 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "9b07bac5ee644116a5772d50a74338b2"
format_stamp: "Formatted at 2026-06-18 23:24:16 on dist-test-slave-47qt"
server_key: "f9c401f602d5201dbf99fb660ea6a41a"
server_key_iv: "a9cb1cf381dbadf0caf9e1e13d250273"
server_key_version: "encryptionkey@0"
I20260618 23:24:16.709034 8888 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:16.725399 8888 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:16.727955 8888 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:16.728138 8888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:16.732409 8888 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:38649
I20260618 23:24:16.732455 8955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:38649 every 8 connection(s)
I20260618 23:24:16.732798 8888 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:24:16.733686 8956 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:16.734905 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 8888
I20260618 23:24:16.734995 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:24:16.735147 8350 external_mini_cluster.cc:1428] Setting key d3ee2bdc28ff0a3795b3d14c248c8e30
I20260618 23:24:16.736269 8956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b07bac5ee644116a5772d50a74338b2: Bootstrap starting.
I20260618 23:24:16.736981 8956 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b07bac5ee644116a5772d50a74338b2: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:16.737341 8956 log.cc:824] T 00000000000000000000000000000000 P 9b07bac5ee644116a5772d50a74338b2: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:16.738099 8956 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b07bac5ee644116a5772d50a74338b2: No bootstrap required, opened a new log
I20260618 23:24:16.740396 8956 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b07bac5ee644116a5772d50a74338b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b07bac5ee644116a5772d50a74338b2" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 38649 } }
I20260618 23:24:16.740571 8956 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b07bac5ee644116a5772d50a74338b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:16.740609 8956 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b07bac5ee644116a5772d50a74338b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b07bac5ee644116a5772d50a74338b2, State: Initialized, Role: FOLLOWER
I20260618 23:24:16.740733 8956 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b07bac5ee644116a5772d50a74338b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b07bac5ee644116a5772d50a74338b2" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 38649 } }
I20260618 23:24:16.740819 8956 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b07bac5ee644116a5772d50a74338b2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:24:16.740851 8956 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b07bac5ee644116a5772d50a74338b2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:24:16.740902 8956 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b07bac5ee644116a5772d50a74338b2 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:16.741683 8956 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b07bac5ee644116a5772d50a74338b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b07bac5ee644116a5772d50a74338b2" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 38649 } }
I20260618 23:24:16.741786 8956 leader_election.cc:302] T 00000000000000000000000000000000 P 9b07bac5ee644116a5772d50a74338b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b07bac5ee644116a5772d50a74338b2; no voters:
I20260618 23:24:16.741971 8956 leader_election.cc:288] T 00000000000000000000000000000000 P 9b07bac5ee644116a5772d50a74338b2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:24:16.742017 8961 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b07bac5ee644116a5772d50a74338b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:16.742164 8961 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b07bac5ee644116a5772d50a74338b2 [term 1 LEADER]: Becoming Leader. State: Replica: 9b07bac5ee644116a5772d50a74338b2, State: Running, Role: LEADER
I20260618 23:24:16.742229 8956 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b07bac5ee644116a5772d50a74338b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:24:16.742270 8961 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b07bac5ee644116a5772d50a74338b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b07bac5ee644116a5772d50a74338b2" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 38649 } }
Jun 18 23:24:16 dist-test-slave-47qt krb5kdc[8875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:16.742723 8962 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b07bac5ee644116a5772d50a74338b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b07bac5ee644116a5772d50a74338b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b07bac5ee644116a5772d50a74338b2" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 38649 } } }
I20260618 23:24:16.742815 8962 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b07bac5ee644116a5772d50a74338b2 [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:16.743002 8970 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:24:16.743041 8963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b07bac5ee644116a5772d50a74338b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b07bac5ee644116a5772d50a74338b2. Latest consensus state: current_term: 1 leader_uuid: "9b07bac5ee644116a5772d50a74338b2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b07bac5ee644116a5772d50a74338b2" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 38649 } } }
I20260618 23:24:16.743201 8963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b07bac5ee644116a5772d50a74338b2 [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:16.744086 8970 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:24:16.746069 8970 catalog_manager.cc:1347] Generated new cluster ID: 6c339b576abf432aa628f5df12859197
I20260618 23:24:16.746115 8970 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:24:16.747031 8959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:16.735680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57802 (local address 127.8.39.190:38649)
0618 23:24:16.735842 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:16.735851 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:16.735891 (+ 40us) server_negotiation.cc:407] Connection header received
0618 23:24:16.736278 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:16.736284 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:16.736352 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:16.736498 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:16.737313 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:16.738546 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:16.739572 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:16.739966 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:16.740054 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:16.742925 (+ 2871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:16.742930 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:16.742930 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:16.742944 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:16.745023 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:16.745462 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:16.745464 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:16.745465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:16.745578 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:16.745820 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:16.745821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:16.745821 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:16.746146 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:16.746287 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:16.746713 (+ 426us) server_negotiation.cc:299] Negotiation successful
0618 23:24:16.746837 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
I20260618 23:24:16.767885 8970 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:24:16.768417 8970 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:24:16.770699 8970 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b07bac5ee644116a5772d50a74338b2: Generated new TSK 0
I20260618 23:24:16.770885 8970 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.39.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
I20260618 23:24:16.796952 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.129:0
--local_ip_for_outbound_sockets=127.8.39.129
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:38649
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:16.879446 8982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:16.879681 8982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:16.879720 8982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:16.880875 8982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:16.880939 8982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:16.880985 8982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.129
I20260618 23:24:16.882707 8982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:38649
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.129
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:16.882985 8982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:16.883220 8982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:16.888474 8987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:16.888474 8990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:16.888502 8988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:16.888924 8982 server_base.cc:1034] running on GCE node
I20260618 23:24:16.889190 8982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:16.889398 8982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:16.890583 8982 hybrid_clock.cc:648] HybridClock initialized: now 1781825056890564 us; error 30 us; skew 500 ppm
Jun 18 23:24:16 dist-test-slave-47qt krb5kdc[8875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825056, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:16.893376 8982 init.cc:377] Logged in from keytab as kudu/127.8.39.129@KRBTEST.COM (short username kudu)
I20260618 23:24:16.893837 8982 webserver.cc:460] Webserver started at http://127.8.39.129:34207/ using document root <none> and password file <none>
I20260618 23:24:16.894074 8982 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:16.894145 8982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:16.894312 8982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:16.895536 8982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-0/data/instance:
uuid: "cc9c134e1032494eb68d238c2034a0af"
format_stamp: "Formatted at 2026-06-18 23:24:16 on dist-test-slave-47qt"
server_key: "61c5d9d2306f9973bb96a33ae69dff13"
server_key_iv: "955a59dca6597453a614085a56866d8f"
server_key_version: "encryptionkey@0"
I20260618 23:24:16.895918 8982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance:
uuid: "cc9c134e1032494eb68d238c2034a0af"
format_stamp: "Formatted at 2026-06-18 23:24:16 on dist-test-slave-47qt"
server_key: "61c5d9d2306f9973bb96a33ae69dff13"
server_key_iv: "955a59dca6597453a614085a56866d8f"
server_key_version: "encryptionkey@0"
I20260618 23:24:16.897414 8982 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:24:16.898157 8997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:16.898334 8982 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260618 23:24:16.898433 8982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-0/wal
uuid: "cc9c134e1032494eb68d238c2034a0af"
format_stamp: "Formatted at 2026-06-18 23:24:16 on dist-test-slave-47qt"
server_key: "61c5d9d2306f9973bb96a33ae69dff13"
server_key_iv: "955a59dca6597453a614085a56866d8f"
server_key_version: "encryptionkey@0"
I20260618 23:24:16.898516 8982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:16.908738 8982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:16.910939 8982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:16.911118 8982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:16.911371 8982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:16.911711 8982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:16.911772 8982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:16.911810 8982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:16.912138 8982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:16.918134 8982 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.129:36141
I20260618 23:24:16.918211 9110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.129:36141 every 8 connection(s)
I20260618 23:24:16.918529 8982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
I20260618 23:24:16.922438 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 8982
I20260618 23:24:16.922531 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance
I20260618 23:24:16.922678 8350 external_mini_cluster.cc:1428] Setting key 4beff3f81a45b35991bc8910ccb7d539
Jun 18 23:24:16 dist-test-slave-47qt krb5kdc[8875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825056, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:16.928812 8959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:16.919245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:38041 (local address 127.8.39.190:38649)
0618 23:24:16.919287 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:16.919289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:16.919482 (+ 193us) server_negotiation.cc:407] Connection header received
0618 23:24:16.920527 (+ 1045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:16.920528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:16.920540 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:16.920595 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:16.921222 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:16.921718 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:16.923014 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:16.923257 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:16.923321 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:16.925689 (+ 2368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:16.925690 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:16.925690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:16.925696 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:16.927139 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:16.927685 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:16.927685 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:16.927686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:16.927734 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:16.928023 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:16.928023 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:16.928023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:16.928122 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:16.928170 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:16.928678 (+ 508us) server_negotiation.cc:299] Negotiation successful
0618 23:24:16.928749 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260618 23:24:16.929060 9113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:16.919204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:38649 (local address 127.8.39.129:38041)
0618 23:24:16.919338 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:16.919385 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:16.920421 (+ 1036us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:16.920659 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:16.920663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:16.920920 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:16.921102 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:16.921105 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:16.921808 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:16.921809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:16.922867 (+ 1058us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:16.922869 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:16.923334 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:16.923334 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:16.923469 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:16.923998 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:16.924007 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:16.925587 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:16.927233 (+ 1646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:16.927240 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:16.927243 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:16.927603 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:16.927801 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:16.927802 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:16.927802 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:16.927950 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:16.928204 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:16.928212 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:16.928478 (+ 266us) client_negotiation.cc:769] Sending connection context
0618 23:24:16.928628 (+ 150us) client_negotiation.cc:240] Negotiation successful
0618 23:24:16.928803 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":63,"threads_started":1}
I20260618 23:24:16.929582 9111 heartbeater.cc:347] Connected to a master server at 127.8.39.190:38649
I20260618 23:24:16.929847 9111 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:16.930075 9111 heartbeater.cc:511] Master 127.8.39.190:38649 requested a full tablet report, sending...
I20260618 23:24:16.930647 8920 ts_manager.cc:194] Registered new tserver with Master: cc9c134e1032494eb68d238c2034a0af (127.8.39.129:36141)
WARNING: no policy specified for kudu/127.8.39.130@KRBTEST.COM; defaulting to no policy
I20260618 23:24:16.931555 8920 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.129@KRBTEST.COM'} at 127.8.39.129:38041
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
I20260618 23:24:16.947638 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.130:0
--local_ip_for_outbound_sockets=127.8.39.130
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:38649
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:17.030471 9116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:17.030706 9116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:17.030745 9116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:17.031929 9116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:17.032004 9116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:17.032048 9116 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.130
I20260618 23:24:17.033610 9116 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:38649
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.130
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:17.033864 9116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:17.034085 9116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:17.038892 9124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:17.038926 9121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:17.039047 9116 server_base.cc:1034] running on GCE node
W20260618 23:24:17.038954 9122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:17.039314 9116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:17.039544 9116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:17.040717 9116 hybrid_clock.cc:648] HybridClock initialized: now 1781825057040695 us; error 50 us; skew 500 ppm
Jun 18 23:24:17 dist-test-slave-47qt krb5kdc[8875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825057, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:17.043665 9116 init.cc:377] Logged in from keytab as kudu/127.8.39.130@KRBTEST.COM (short username kudu)
I20260618 23:24:17.044174 9116 webserver.cc:460] Webserver started at http://127.8.39.130:45733/ using document root <none> and password file <none>
I20260618 23:24:17.044413 9116 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:17.044476 9116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:17.044605 9116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:17.046087 9116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-1/data/instance:
uuid: "9cb72b6b44ad42e8842bed2baeeb1e76"
format_stamp: "Formatted at 2026-06-18 23:24:17 on dist-test-slave-47qt"
server_key: "f06dff7e6003bb8d7542aa5d39129eb2"
server_key_iv: "72ccc5b2cab52144192283018a2ffb55"
server_key_version: "encryptionkey@0"
I20260618 23:24:17.046458 9116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance:
uuid: "9cb72b6b44ad42e8842bed2baeeb1e76"
format_stamp: "Formatted at 2026-06-18 23:24:17 on dist-test-slave-47qt"
server_key: "f06dff7e6003bb8d7542aa5d39129eb2"
server_key_iv: "72ccc5b2cab52144192283018a2ffb55"
server_key_version: "encryptionkey@0"
I20260618 23:24:17.047715 9116 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:24:17.048642 9131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:17.049027 9116 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:17.049328 9116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-1/wal
uuid: "9cb72b6b44ad42e8842bed2baeeb1e76"
format_stamp: "Formatted at 2026-06-18 23:24:17 on dist-test-slave-47qt"
server_key: "f06dff7e6003bb8d7542aa5d39129eb2"
server_key_iv: "72ccc5b2cab52144192283018a2ffb55"
server_key_version: "encryptionkey@0"
I20260618 23:24:17.049443 9116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:17.067209 9116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:17.070171 9116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:17.070351 9116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:17.070601 9116 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:17.070915 9116 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:17.070976 9116 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:17.071018 9116 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:17.071046 9116 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:17.077368 9116 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.130:40613
I20260618 23:24:17.077405 9244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.130:40613 every 8 connection(s)
I20260618 23:24:17.077756 9116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
I20260618 23:24:17.082888 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 9116
I20260618 23:24:17.083014 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance
I20260618 23:24:17.083169 8350 external_mini_cluster.cc:1428] Setting key da47d5544a2991a75f6880771338b498
Jun 18 23:24:17 dist-test-slave-47qt krb5kdc[8875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825057, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:17.088228 8959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:17.078555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:53437 (local address 127.8.39.190:38649)
0618 23:24:17.078686 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:17.078688 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:17.078889 (+ 201us) server_negotiation.cc:407] Connection header received
0618 23:24:17.079863 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:17.079864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:17.079876 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:17.079930 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:17.080568 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.081068 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.082117 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.082335 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.082425 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:17.085328 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:17.085329 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:17.085330 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:17.085337 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:17.086805 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:17.087248 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:17.087250 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:17.087250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:17.087310 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:17.087567 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:17.087568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:17.087569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:17.087682 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:17.087765 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:17.088051 (+ 286us) server_negotiation.cc:299] Negotiation successful
0618 23:24:17.088141 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260618 23:24:17.088423 9247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:17.078509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:38649 (local address 127.8.39.130:53437)
0618 23:24:17.078722 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:17.078798 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:17.079740 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:17.080007 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:17.080011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:17.080279 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:17.080463 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:17.080466 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.081195 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.081196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:17.082009 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:17.082011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.082430 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.082432 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:17.082558 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:17.083141 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:17.083149 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:17.085220 (+ 2071us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:17.086906 (+ 1686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:17.086913 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:17.086915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:17.087150 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:17.087377 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:17.087378 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:17.087378 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:17.087490 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:17.087749 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:17.087755 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:17.087899 (+ 144us) client_negotiation.cc:769] Sending connection context
0618 23:24:17.088013 (+ 114us) client_negotiation.cc:240] Negotiation successful
0618 23:24:17.088172 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":92,"threads_started":1}
I20260618 23:24:17.088968 9245 heartbeater.cc:347] Connected to a master server at 127.8.39.190:38649
I20260618 23:24:17.089074 9245 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:17.089285 9245 heartbeater.cc:511] Master 127.8.39.190:38649 requested a full tablet report, sending...
I20260618 23:24:17.089671 8920 ts_manager.cc:194] Registered new tserver with Master: 9cb72b6b44ad42e8842bed2baeeb1e76 (127.8.39.130:40613)
I20260618 23:24:17.090052 8920 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.130@KRBTEST.COM'} at 127.8.39.130:53437
WARNING: no policy specified for kudu/127.8.39.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
I20260618 23:24:17.108959 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.131:0
--local_ip_for_outbound_sockets=127.8.39.131
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:38649
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:41357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:17.192775 9250 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:17.192978 9250 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:17.193003 9250 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:17.194192 9250 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:17.194234 9250 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:17.194262 9250 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.131
I20260618 23:24:17.195765 9250 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:41357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:38649
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9250
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.131
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:17.196017 9250 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:17.196241 9250 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:17.201717 9256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:17.201781 9255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:17.201797 9258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:17.201858 9250 server_base.cc:1034] running on GCE node
I20260618 23:24:17.202173 9250 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:17.202417 9250 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:17.203622 9250 hybrid_clock.cc:648] HybridClock initialized: now 1781825057203596 us; error 38 us; skew 500 ppm
Jun 18 23:24:17 dist-test-slave-47qt krb5kdc[8875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825057, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:17.206403 9250 init.cc:377] Logged in from keytab as kudu/127.8.39.131@KRBTEST.COM (short username kudu)
I20260618 23:24:17.206832 9250 webserver.cc:460] Webserver started at http://127.8.39.131:37529/ using document root <none> and password file <none>
I20260618 23:24:17.207068 9250 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:17.207135 9250 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:17.207293 9250 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:17.208349 9250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-2/data/instance:
uuid: "80c80cdf7d554b28b793a0857c5d4fb7"
format_stamp: "Formatted at 2026-06-18 23:24:17 on dist-test-slave-47qt"
server_key: "9d64185dcdbee6b4210b579a7ff13833"
server_key_iv: "6813931f55c4ff48e26f4461dc9d554d"
server_key_version: "encryptionkey@0"
I20260618 23:24:17.208822 9250 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance:
uuid: "80c80cdf7d554b28b793a0857c5d4fb7"
format_stamp: "Formatted at 2026-06-18 23:24:17 on dist-test-slave-47qt"
server_key: "9d64185dcdbee6b4210b579a7ff13833"
server_key_iv: "6813931f55c4ff48e26f4461dc9d554d"
server_key_version: "encryptionkey@0"
I20260618 23:24:17.210470 9250 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:24:17.211548 9265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:17.211740 9250 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:17.212083 9250 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-2/wal
uuid: "80c80cdf7d554b28b793a0857c5d4fb7"
format_stamp: "Formatted at 2026-06-18 23:24:17 on dist-test-slave-47qt"
server_key: "9d64185dcdbee6b4210b579a7ff13833"
server_key_iv: "6813931f55c4ff48e26f4461dc9d554d"
server_key_version: "encryptionkey@0"
I20260618 23:24:17.212218 9250 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:17.230195 9250 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:17.233485 9250 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:17.233685 9250 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:17.233938 9250 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:17.234478 9250 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:17.234534 9250 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:17.234558 9250 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:17.234571 9250 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:17.240414 9250 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.131:40597
I20260618 23:24:17.240461 9378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.131:40597 every 8 connection(s)
I20260618 23:24:17.240784 9250 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
I20260618 23:24:17.244422 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 9250
I20260618 23:24:17.244546 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance
I20260618 23:24:17.244743 8350 external_mini_cluster.cc:1428] Setting key b74e3277e794cc9e0b217db055db1219
Jun 18 23:24:17 dist-test-slave-47qt krb5kdc[8875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825057, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:17.252351 8959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:17.241662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:54643 (local address 127.8.39.190:38649)
0618 23:24:17.241730 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:17.241732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:17.241964 (+ 232us) server_negotiation.cc:407] Connection header received
0618 23:24:17.243142 (+ 1178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:17.243144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:17.243156 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:17.243230 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:17.243951 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.244550 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.246367 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.246561 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.246676 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:17.249438 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:17.249440 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:17.249440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:17.249454 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:17.250859 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:17.251310 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:17.251312 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:17.251312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:17.251379 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:17.251631 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:17.251632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:17.251632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:17.251762 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:17.251839 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:17.252157 (+ 318us) server_negotiation.cc:299] Negotiation successful
0618 23:24:17.252234 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260618 23:24:17.252540 9381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:17.241603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:38649 (local address 127.8.39.131:54643)
0618 23:24:17.241857 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:17.241894 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:17.242926 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:17.243319 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:17.243322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:17.243657 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:17.243852 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:17.243856 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.244665 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.244666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:17.246262 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:17.246264 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.246689 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.246690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:17.246820 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:17.247411 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:17.247420 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:17.249321 (+ 1901us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:17.250995 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:17.251002 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:17.251005 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:17.251230 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:17.251449 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:17.251450 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:17.251450 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:17.251556 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:17.251838 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:17.251845 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:17.252007 (+ 162us) client_negotiation.cc:769] Sending connection context
0618 23:24:17.252137 (+ 130us) client_negotiation.cc:240] Negotiation successful
0618 23:24:17.252303 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":107,"threads_started":1}
I20260618 23:24:17.253026 9379 heartbeater.cc:347] Connected to a master server at 127.8.39.190:38649
I20260618 23:24:17.253170 9379 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:17.253377 9379 heartbeater.cc:511] Master 127.8.39.190:38649 requested a full tablet report, sending...
I20260618 23:24:17.253813 8920 ts_manager.cc:194] Registered new tserver with Master: 80c80cdf7d554b28b793a0857c5d4fb7 (127.8.39.131:40597)
I20260618 23:24:17.254187 8920 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.131@KRBTEST.COM'} at 127.8.39.131:54643
I20260618 23:24:17.257618 8350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:24:17.266319 8959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:17.259126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57814 (local address 127.8.39.190:38649)
0618 23:24:17.259189 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:17.259191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:17.259276 (+ 85us) server_negotiation.cc:407] Connection header received
0618 23:24:17.259340 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:17.259341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:17.259351 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:17.259423 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:17.259957 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.260439 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.261167 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.261361 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.261448 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:17.262632 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:17.262634 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:17.262634 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:17.262641 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:17.264565 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:17.265063 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:17.265066 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:17.265066 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:17.265243 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:17.265683 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:17.265685 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:17.265685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:17.265869 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:17.265990 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:17.266123 (+ 133us) server_negotiation.cc:299] Negotiation successful
0618 23:24:17.266216 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260618 23:24:17.268896 8920 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:24:17.271596 8920 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:24:17.278204 9393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:17.274150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:40613 (local address 127.0.0.1:40112)
0618 23:24:17.274466 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:17.274478 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:17.274539 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:17.274828 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:17.274831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:17.274833 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:17.274906 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:17.274908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.275858 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.275859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:17.277096 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:17.277099 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.277895 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.277896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:17.277974 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:17.277987 (+ 13us) client_negotiation.cc:769] Sending connection context
0618 23:24:17.278052 (+ 65us) client_negotiation.cc:240] Negotiation successful
0618 23:24:17.278114 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":297,"mutex_wait_us":2,"thread_start_us":82,"threads_started":1}
I20260618 23:24:17.278267 9392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:17.274242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41254 (local address 127.8.39.131:40597)
0618 23:24:17.274674 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:17.274678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:17.274686 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:24:17.274720 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:17.274722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:17.274736 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:17.275002 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:17.275072 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.276103 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.277208 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.277897 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.278004 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:17.278058 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:17.278132 (+ 74us) server_negotiation.cc:299] Negotiation successful
0618 23:24:17.278180 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":88,"threads_started":1}
I20260618 23:24:17.278494 9391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:17.274342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:36141 (local address 127.0.0.1:59254)
0618 23:24:17.274509 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:17.274518 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:17.274560 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:17.274889 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:17.274890 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:17.274891 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:17.274925 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:17.274928 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.276221 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.276222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:17.277331 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:17.277344 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.278259 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.278260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:17.278328 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:17.278332 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:24:17.278394 (+ 62us) client_negotiation.cc:240] Negotiation successful
0618 23:24:17.278446 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":33,"thread_start_us":70,"threads_started":1}
I20260618 23:24:17.278545 9394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:17.274347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40112 (local address 127.8.39.130:40613)
0618 23:24:17.274679 (+ 332us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:17.274682 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:17.274690 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:24:17.274719 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:17.274720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:17.274733 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:17.274822 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:17.274992 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.275748 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.277209 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.277781 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.277890 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:17.277953 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:17.278435 (+ 482us) server_negotiation.cc:299] Negotiation successful
0618 23:24:17.278476 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":83,"threads_started":1}
I20260618 23:24:17.278806 9395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:17.274485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59254 (local address 127.8.39.129:36141)
0618 23:24:17.274665 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:17.274670 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:17.274680 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:24:17.274720 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:17.274722 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:17.274736 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:17.274813 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:17.275006 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.276119 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.277427 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.278120 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.278181 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:17.278227 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:17.278676 (+ 449us) server_negotiation.cc:299] Negotiation successful
0618 23:24:17.278730 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":77,"threads_started":1}
I20260618 23:24:17.278205 9390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:17.274150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:40597 (local address 127.0.0.1:41254)
0618 23:24:17.274420 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:17.274461 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:17.274540 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:17.274853 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:17.274854 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:17.274857 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:17.274925 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:17.274927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.276235 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.276236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:17.277103 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:17.277105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.278006 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.278007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:17.278068 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:17.278071 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:24:17.278121 (+ 50us) client_negotiation.cc:240] Negotiation successful
0618 23:24:17.278150 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":51,"threads_started":1}
I20260618 23:24:17.279330 9179 tablet_service.cc:1467] Processing CreateTablet for tablet 08c9d763c51b41f389f6792d21e74b00 (DEFAULT_TABLE table=test-table [id=c27dfd035d724aa884b20837cdd75f0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:17.279456 9045 tablet_service.cc:1467] Processing CreateTablet for tablet 08c9d763c51b41f389f6792d21e74b00 (DEFAULT_TABLE table=test-table [id=c27dfd035d724aa884b20837cdd75f0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:17.279495 9313 tablet_service.cc:1467] Processing CreateTablet for tablet 08c9d763c51b41f389f6792d21e74b00 (DEFAULT_TABLE table=test-table [id=c27dfd035d724aa884b20837cdd75f0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:17.279711 9179 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08c9d763c51b41f389f6792d21e74b00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:17.279717 9045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08c9d763c51b41f389f6792d21e74b00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:17.279852 9313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 08c9d763c51b41f389f6792d21e74b00. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:17.282528 9396 tablet_bootstrap.cc:492] T 08c9d763c51b41f389f6792d21e74b00 P 80c80cdf7d554b28b793a0857c5d4fb7: Bootstrap starting.
I20260618 23:24:17.282948 9397 tablet_bootstrap.cc:492] T 08c9d763c51b41f389f6792d21e74b00 P 9cb72b6b44ad42e8842bed2baeeb1e76: Bootstrap starting.
I20260618 23:24:17.283183 9398 tablet_bootstrap.cc:492] T 08c9d763c51b41f389f6792d21e74b00 P cc9c134e1032494eb68d238c2034a0af: Bootstrap starting.
I20260618 23:24:17.283223 9396 tablet_bootstrap.cc:654] T 08c9d763c51b41f389f6792d21e74b00 P 80c80cdf7d554b28b793a0857c5d4fb7: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:17.283670 9397 tablet_bootstrap.cc:654] T 08c9d763c51b41f389f6792d21e74b00 P 9cb72b6b44ad42e8842bed2baeeb1e76: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:17.283739 9398 tablet_bootstrap.cc:654] T 08c9d763c51b41f389f6792d21e74b00 P cc9c134e1032494eb68d238c2034a0af: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:17.283821 9396 log.cc:824] T 08c9d763c51b41f389f6792d21e74b00 P 80c80cdf7d554b28b793a0857c5d4fb7: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:17.284212 9397 log.cc:824] T 08c9d763c51b41f389f6792d21e74b00 P 9cb72b6b44ad42e8842bed2baeeb1e76: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:17.284432 9398 log.cc:824] T 08c9d763c51b41f389f6792d21e74b00 P cc9c134e1032494eb68d238c2034a0af: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:17.284816 9396 tablet_bootstrap.cc:492] T 08c9d763c51b41f389f6792d21e74b00 P 80c80cdf7d554b28b793a0857c5d4fb7: No bootstrap required, opened a new log
I20260618 23:24:17.284906 9396 ts_tablet_manager.cc:1397] T 08c9d763c51b41f389f6792d21e74b00 P 80c80cdf7d554b28b793a0857c5d4fb7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260618 23:24:17.284927 9397 tablet_bootstrap.cc:492] T 08c9d763c51b41f389f6792d21e74b00 P 9cb72b6b44ad42e8842bed2baeeb1e76: No bootstrap required, opened a new log
I20260618 23:24:17.284987 9397 ts_tablet_manager.cc:1397] T 08c9d763c51b41f389f6792d21e74b00 P 9cb72b6b44ad42e8842bed2baeeb1e76: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260618 23:24:17.285310 9398 tablet_bootstrap.cc:492] T 08c9d763c51b41f389f6792d21e74b00 P cc9c134e1032494eb68d238c2034a0af: No bootstrap required, opened a new log
I20260618 23:24:17.285372 9398 ts_tablet_manager.cc:1397] T 08c9d763c51b41f389f6792d21e74b00 P cc9c134e1032494eb68d238c2034a0af: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260618 23:24:17.286566 9397 raft_consensus.cc:348] T 08c9d763c51b41f389f6792d21e74b00 P 9cb72b6b44ad42e8842bed2baeeb1e76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb72b6b44ad42e8842bed2baeeb1e76" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 40613 } } peers { permanent_uuid: "cc9c134e1032494eb68d238c2034a0af" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 36141 } } peers { permanent_uuid: "80c80cdf7d554b28b793a0857c5d4fb7" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 40597 } }
I20260618 23:24:17.286734 9397 raft_consensus.cc:374] T 08c9d763c51b41f389f6792d21e74b00 P 9cb72b6b44ad42e8842bed2baeeb1e76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:17.286777 9397 raft_consensus.cc:729] T 08c9d763c51b41f389f6792d21e74b00 P 9cb72b6b44ad42e8842bed2baeeb1e76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cb72b6b44ad42e8842bed2baeeb1e76, State: Initialized, Role: FOLLOWER
I20260618 23:24:17.286890 9397 consensus_queue.cc:260] T 08c9d763c51b41f389f6792d21e74b00 P 9cb72b6b44ad42e8842bed2baeeb1e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb72b6b44ad42e8842bed2baeeb1e76" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 40613 } } peers { permanent_uuid: "cc9c134e1032494eb68d238c2034a0af" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 36141 } } peers { permanent_uuid: "80c80cdf7d554b28b793a0857c5d4fb7" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 40597 } }
I20260618 23:24:17.286901 9396 raft_consensus.cc:348] T 08c9d763c51b41f389f6792d21e74b00 P 80c80cdf7d554b28b793a0857c5d4fb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb72b6b44ad42e8842bed2baeeb1e76" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 40613 } } peers { permanent_uuid: "cc9c134e1032494eb68d238c2034a0af" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 36141 } } peers { permanent_uuid: "80c80cdf7d554b28b793a0857c5d4fb7" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 40597 } }
I20260618 23:24:17.287074 9396 raft_consensus.cc:374] T 08c9d763c51b41f389f6792d21e74b00 P 80c80cdf7d554b28b793a0857c5d4fb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:17.287120 9396 raft_consensus.cc:729] T 08c9d763c51b41f389f6792d21e74b00 P 80c80cdf7d554b28b793a0857c5d4fb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80c80cdf7d554b28b793a0857c5d4fb7, State: Initialized, Role: FOLLOWER
I20260618 23:24:17.287200 9397 ts_tablet_manager.cc:1428] T 08c9d763c51b41f389f6792d21e74b00 P 9cb72b6b44ad42e8842bed2baeeb1e76: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260618 23:24:17.287266 9245 heartbeater.cc:503] Master 127.8.39.190:38649 was elected leader, sending a full tablet report...
I20260618 23:24:17.287249 9396 consensus_queue.cc:260] T 08c9d763c51b41f389f6792d21e74b00 P 80c80cdf7d554b28b793a0857c5d4fb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb72b6b44ad42e8842bed2baeeb1e76" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 40613 } } peers { permanent_uuid: "cc9c134e1032494eb68d238c2034a0af" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 36141 } } peers { permanent_uuid: "80c80cdf7d554b28b793a0857c5d4fb7" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 40597 } }
I20260618 23:24:17.287294 9398 raft_consensus.cc:348] T 08c9d763c51b41f389f6792d21e74b00 P cc9c134e1032494eb68d238c2034a0af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb72b6b44ad42e8842bed2baeeb1e76" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 40613 } } peers { permanent_uuid: "cc9c134e1032494eb68d238c2034a0af" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 36141 } } peers { permanent_uuid: "80c80cdf7d554b28b793a0857c5d4fb7" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 40597 } }
I20260618 23:24:17.287456 9398 raft_consensus.cc:374] T 08c9d763c51b41f389f6792d21e74b00 P cc9c134e1032494eb68d238c2034a0af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:17.287494 9398 raft_consensus.cc:729] T 08c9d763c51b41f389f6792d21e74b00 P cc9c134e1032494eb68d238c2034a0af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc9c134e1032494eb68d238c2034a0af, State: Initialized, Role: FOLLOWER
I20260618 23:24:17.287509 9396 ts_tablet_manager.cc:1428] T 08c9d763c51b41f389f6792d21e74b00 P 80c80cdf7d554b28b793a0857c5d4fb7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:24:17.287532 9379 heartbeater.cc:503] Master 127.8.39.190:38649 was elected leader, sending a full tablet report...
I20260618 23:24:17.287611 9398 consensus_queue.cc:260] T 08c9d763c51b41f389f6792d21e74b00 P cc9c134e1032494eb68d238c2034a0af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb72b6b44ad42e8842bed2baeeb1e76" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 40613 } } peers { permanent_uuid: "cc9c134e1032494eb68d238c2034a0af" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 36141 } } peers { permanent_uuid: "80c80cdf7d554b28b793a0857c5d4fb7" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 40597 } }
I20260618 23:24:17.287887 9398 ts_tablet_manager.cc:1428] T 08c9d763c51b41f389f6792d21e74b00 P cc9c134e1032494eb68d238c2034a0af: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260618 23:24:17.287897 9111 heartbeater.cc:503] Master 127.8.39.190:38649 was elected leader, sending a full tablet report...
W20260618 23:24:17.328410 9246 tablet.cc:2369] T 08c9d763c51b41f389f6792d21e74b00 P 9cb72b6b44ad42e8842bed2baeeb1e76: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:24:17.369315 9403 raft_consensus.cc:482] T 08c9d763c51b41f389f6792d21e74b00 P 80c80cdf7d554b28b793a0857c5d4fb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:24:17.369428 9403 raft_consensus.cc:504] T 08c9d763c51b41f389f6792d21e74b00 P 80c80cdf7d554b28b793a0857c5d4fb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb72b6b44ad42e8842bed2baeeb1e76" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 40613 } } peers { permanent_uuid: "cc9c134e1032494eb68d238c2034a0af" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 36141 } } peers { permanent_uuid: "80c80cdf7d554b28b793a0857c5d4fb7" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 40597 } }
I20260618 23:24:17.369711 9403 leader_election.cc:288] T 08c9d763c51b41f389f6792d21e74b00 P 80c80cdf7d554b28b793a0857c5d4fb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9cb72b6b44ad42e8842bed2baeeb1e76 (127.8.39.130:40613), cc9c134e1032494eb68d238c2034a0af (127.8.39.129:36141)
I20260618 23:24:17.372716 9405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:17.370075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:36141 (local address 127.8.39.131:41903)
0618 23:24:17.370254 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:17.370264 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:17.370312 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:17.370481 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:17.370482 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:17.370483 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:17.370552 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:17.370553 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.371175 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.371176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:17.371917 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:17.371918 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.372458 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.372459 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:17.372528 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:17.372534 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 23:24:17.372581 (+ 47us) client_negotiation.cc:240] Negotiation successful
0618 23:24:17.372610 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":102,"threads_started":1}
I20260618 23:24:17.372728 9395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:17.370225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:41903 (local address 127.8.39.129:36141)
0618 23:24:17.370293 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:17.370295 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:17.370304 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:24:17.370368 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:17.370368 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:17.370380 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:17.370440 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:17.370627 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.371065 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.372018 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.372373 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.372430 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:17.372460 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:17.372616 (+ 156us) server_negotiation.cc:299] Negotiation successful
0618 23:24:17.372663 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260618 23:24:17.372793 9394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:17.370087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:58725 (local address 127.8.39.130:40613)
0618 23:24:17.370154 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:17.370160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:17.370168 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:24:17.370215 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:17.370216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:17.370230 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:17.370316 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:17.370497 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.371052 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.372015 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.372386 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.372474 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:17.372481 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:17.372678 (+ 197us) server_negotiation.cc:299] Negotiation successful
0618 23:24:17.372713 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260618 23:24:17.372829 9381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:17.369955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:40613 (local address 127.8.39.131:58725)
0618 23:24:17.370014 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:17.370027 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:17.370070 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:17.370329 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:17.370330 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:17.370332 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:17.370420 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:17.370423 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.371132 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.371133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:17.371912 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:17.371915 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.372487 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.372488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:17.372589 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:17.372593 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:24:17.372651 (+ 58us) client_negotiation.cc:240] Negotiation successful
0618 23:24:17.372695 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":10}
I20260618 23:24:17.373093 9065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08c9d763c51b41f389f6792d21e74b00" candidate_uuid: "80c80cdf7d554b28b793a0857c5d4fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc9c134e1032494eb68d238c2034a0af" is_pre_election: true
I20260618 23:24:17.373198 9199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08c9d763c51b41f389f6792d21e74b00" candidate_uuid: "80c80cdf7d554b28b793a0857c5d4fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cb72b6b44ad42e8842bed2baeeb1e76" is_pre_election: true
I20260618 23:24:17.373286 9065 raft_consensus.cc:2454] T 08c9d763c51b41f389f6792d21e74b00 P cc9c134e1032494eb68d238c2034a0af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80c80cdf7d554b28b793a0857c5d4fb7 in term 0.
I20260618 23:24:17.373364 9199 raft_consensus.cc:2454] T 08c9d763c51b41f389f6792d21e74b00 P 9cb72b6b44ad42e8842bed2baeeb1e76 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 80c80cdf7d554b28b793a0857c5d4fb7 in term 0.
I20260618 23:24:17.373538 9266 leader_election.cc:302] T 08c9d763c51b41f389f6792d21e74b00 P 80c80cdf7d554b28b793a0857c5d4fb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80c80cdf7d554b28b793a0857c5d4fb7, cc9c134e1032494eb68d238c2034a0af; no voters:
I20260618 23:24:17.373689 9403 raft_consensus.cc:2781] T 08c9d763c51b41f389f6792d21e74b00 P 80c80cdf7d554b28b793a0857c5d4fb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:24:17.373759 9403 raft_consensus.cc:482] T 08c9d763c51b41f389f6792d21e74b00 P 80c80cdf7d554b28b793a0857c5d4fb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:24:17.373777 9403 raft_consensus.cc:3037] T 08c9d763c51b41f389f6792d21e74b00 P 80c80cdf7d554b28b793a0857c5d4fb7 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:17.374583 9403 raft_consensus.cc:504] T 08c9d763c51b41f389f6792d21e74b00 P 80c80cdf7d554b28b793a0857c5d4fb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb72b6b44ad42e8842bed2baeeb1e76" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 40613 } } peers { permanent_uuid: "cc9c134e1032494eb68d238c2034a0af" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 36141 } } peers { permanent_uuid: "80c80cdf7d554b28b793a0857c5d4fb7" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 40597 } }
I20260618 23:24:17.374712 9403 leader_election.cc:288] T 08c9d763c51b41f389f6792d21e74b00 P 80c80cdf7d554b28b793a0857c5d4fb7 [CANDIDATE]: Term 1 election: Requested vote from peers 9cb72b6b44ad42e8842bed2baeeb1e76 (127.8.39.130:40613), cc9c134e1032494eb68d238c2034a0af (127.8.39.129:36141)
I20260618 23:24:17.374833 9199 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08c9d763c51b41f389f6792d21e74b00" candidate_uuid: "80c80cdf7d554b28b793a0857c5d4fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9cb72b6b44ad42e8842bed2baeeb1e76"
I20260618 23:24:17.374914 9199 raft_consensus.cc:3037] T 08c9d763c51b41f389f6792d21e74b00 P 9cb72b6b44ad42e8842bed2baeeb1e76 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:17.374892 9065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "08c9d763c51b41f389f6792d21e74b00" candidate_uuid: "80c80cdf7d554b28b793a0857c5d4fb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc9c134e1032494eb68d238c2034a0af"
I20260618 23:24:17.374987 9065 raft_consensus.cc:3037] T 08c9d763c51b41f389f6792d21e74b00 P cc9c134e1032494eb68d238c2034a0af [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:17.375581 9199 raft_consensus.cc:2454] T 08c9d763c51b41f389f6792d21e74b00 P 9cb72b6b44ad42e8842bed2baeeb1e76 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80c80cdf7d554b28b793a0857c5d4fb7 in term 1.
I20260618 23:24:17.375713 9065 raft_consensus.cc:2454] T 08c9d763c51b41f389f6792d21e74b00 P cc9c134e1032494eb68d238c2034a0af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 80c80cdf7d554b28b793a0857c5d4fb7 in term 1.
I20260618 23:24:17.375736 9266 leader_election.cc:302] T 08c9d763c51b41f389f6792d21e74b00 P 80c80cdf7d554b28b793a0857c5d4fb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 80c80cdf7d554b28b793a0857c5d4fb7, 9cb72b6b44ad42e8842bed2baeeb1e76; no voters:
I20260618 23:24:17.375809 9403 raft_consensus.cc:2781] T 08c9d763c51b41f389f6792d21e74b00 P 80c80cdf7d554b28b793a0857c5d4fb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:17.375960 9403 raft_consensus.cc:686] T 08c9d763c51b41f389f6792d21e74b00 P 80c80cdf7d554b28b793a0857c5d4fb7 [term 1 LEADER]: Becoming Leader. State: Replica: 80c80cdf7d554b28b793a0857c5d4fb7, State: Running, Role: LEADER
I20260618 23:24:17.376065 9403 consensus_queue.cc:237] T 08c9d763c51b41f389f6792d21e74b00 P 80c80cdf7d554b28b793a0857c5d4fb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb72b6b44ad42e8842bed2baeeb1e76" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 40613 } } peers { permanent_uuid: "cc9c134e1032494eb68d238c2034a0af" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 36141 } } peers { permanent_uuid: "80c80cdf7d554b28b793a0857c5d4fb7" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 40597 } }
I20260618 23:24:17.376859 8920 catalog_manager.cc:5495] T 08c9d763c51b41f389f6792d21e74b00 P 80c80cdf7d554b28b793a0857c5d4fb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 80c80cdf7d554b28b793a0857c5d4fb7 (127.8.39.131). New cstate: current_term: 1 leader_uuid: "80c80cdf7d554b28b793a0857c5d4fb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9cb72b6b44ad42e8842bed2baeeb1e76" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 40613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc9c134e1032494eb68d238c2034a0af" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 36141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "80c80cdf7d554b28b793a0857c5d4fb7" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 40597 } health_report { overall_health: HEALTHY } } }
Jun 18 23:24:17 dist-test-slave-47qt krb5kdc[8875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825056, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.129@KRBTEST.COM
I20260618 23:24:17.400054 9395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:17.391346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59270 (local address 127.8.39.129:36141)
0618 23:24:17.391496 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:17.391498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:17.391538 (+ 40us) server_negotiation.cc:407] Connection header received
0618 23:24:17.391567 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:17.391568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:17.391581 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:17.391630 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:17.392293 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.392780 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.393660 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.393844 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.393916 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:17.396435 (+ 2519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:17.396445 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:17.396445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:17.396453 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:17.398269 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:17.398652 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:17.398655 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:17.398655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:17.398711 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:17.399120 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:17.399121 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:17.399122 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:17.399292 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:17.399425 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:17.399695 (+ 270us) server_negotiation.cc:299] Negotiation successful
0618 23:24:17.399903 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
Jun 18 23:24:17 dist-test-slave-47qt krb5kdc[8875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825057, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260618 23:24:17.419256 9112 tablet.cc:2369] T 08c9d763c51b41f389f6792d21e74b00 P cc9c134e1032494eb68d238c2034a0af: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 18 23:24:17 dist-test-slave-47qt krb5kdc[8875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825057, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.39.129@KRBTEST.COM
I20260618 23:24:17.425726 9395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:17.418445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59278 (local address 127.8.39.129:36141)
0618 23:24:17.418513 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:17.418516 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:17.418595 (+ 79us) server_negotiation.cc:407] Connection header received
0618 23:24:17.418695 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:17.418696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:17.418713 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:17.418776 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:17.419419 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.419904 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.420596 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.420779 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.420835 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:17.422974 (+ 2139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:17.422975 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:17.422975 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:17.422982 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:17.424522 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:17.424887 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:17.424888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:17.424889 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:17.424953 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:17.425239 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:17.425240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:17.425240 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:17.425334 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:17.425422 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:17.425582 (+ 160us) server_negotiation.cc:299] Negotiation successful
0618 23:24:17.425643 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
Jun 18 23:24:17 dist-test-slave-47qt krb5kdc[8875](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 18 23:24:17 dist-test-slave-47qt krb5kdc[8875](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.129@KRBTEST.COM
I20260618 23:24:17.449338 9395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:17.440945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59280 (local address 127.8.39.129:36141)
0618 23:24:17.441031 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:17.441033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:17.441044 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:24:17.441257 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:17.441258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:17.441275 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:17.441320 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:17.441921 (+ 601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.442413 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.443229 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:17.443418 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:17.443481 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:17.446260 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:17.446262 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:17.446262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:17.446268 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:17.447706 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:17.448263 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:17.448265 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:17.448266 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:17.448344 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:17.448652 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:17.448653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:17.448654 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:17.448792 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:17.448887 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:17.449096 (+ 209us) server_negotiation.cc:299] Negotiation successful
0618 23:24:17.449234 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
W20260618 23:24:17.450094 9025 tablet_service.cc:2908] Rejecting scan request for tablet 08c9d763c51b41f389f6792d21e74b00: Uninitialized: clean time has not yet been initialized
I20260618 23:24:17.451078 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 8982
I20260618 23:24:17.456166 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 9116
I20260618 23:24:17.460947 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 9250
I20260618 23:24:17.466046 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 8888
2026-06-18T23:24:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2028 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:24:17 dist-test-slave-47qt krb5kdc[9420](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:24:17 dist-test-slave-47qt krb5kdc[9420](info): set up 2 sockets
Jun 18 23:24:17 dist-test-slave-47qt krb5kdc[9420](info): commencing operation
krb5kdc: starting...
W20260618 23:24:18.532605 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:24:18 dist-test-slave-47qt krb5kdc[9420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:24:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:24:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.39.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
I20260618 23:24:18.656054 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:34303
--webserver_interface=127.8.39.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:44985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:34303
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:18.736670 9433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:18.736876 9433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:18.736898 9433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:18.738109 9433 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:24:18.738168 9433 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260618 23:24:18.738227 9433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:18.738247 9433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:18.738272 9433 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:24:18.738299 9433 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:24:18.739768 9433 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:44985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:34303
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:34303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:18.740020 9433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:18.740255 9433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:18.745199 9441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:18.745220 9438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:18.745270 9439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:18.745445 9433 server_base.cc:1034] running on GCE node
I20260618 23:24:18.745685 9433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:18.746004 9433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:18.747185 9433 hybrid_clock.cc:648] HybridClock initialized: now 1781825058747190 us; error 47 us; skew 500 ppm
Jun 18 23:24:18 dist-test-slave-47qt krb5kdc[9420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825058, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:18.753747 9433 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:24:18.754156 9433 webserver.cc:460] Webserver started at http://127.8.39.190:40483/ using document root <none> and password file <none>
I20260618 23:24:18.754416 9433 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:18.754462 9433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:18.754594 9433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:18.755681 9433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "9ae35dc6bac64997a96505c30e9e6860"
format_stamp: "Formatted at 2026-06-18 23:24:18 on dist-test-slave-47qt"
server_key: "38b4374192c8f6d018a79d94fb47fe4f"
server_key_iv: "eedeb6dc6a5bf43b85af19116418b163"
server_key_version: "encryptionkey@0"
I20260618 23:24:18.756162 9433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "9ae35dc6bac64997a96505c30e9e6860"
format_stamp: "Formatted at 2026-06-18 23:24:18 on dist-test-slave-47qt"
server_key: "38b4374192c8f6d018a79d94fb47fe4f"
server_key_iv: "eedeb6dc6a5bf43b85af19116418b163"
server_key_version: "encryptionkey@0"
I20260618 23:24:18.757602 9433 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:24:18.758396 9448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:18.758597 9433 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:18.758697 9433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "9ae35dc6bac64997a96505c30e9e6860"
format_stamp: "Formatted at 2026-06-18 23:24:18 on dist-test-slave-47qt"
server_key: "38b4374192c8f6d018a79d94fb47fe4f"
server_key_iv: "eedeb6dc6a5bf43b85af19116418b163"
server_key_version: "encryptionkey@0"
I20260618 23:24:18.758778 9433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:18.782863 9433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:18.785250 9433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:18.785400 9433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:18.790896 9433 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:34303
I20260618 23:24:18.790935 9510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:34303 every 8 connection(s)
I20260618 23:24:18.791261 9433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:24:18.791857 9511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:18.794062 9511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ae35dc6bac64997a96505c30e9e6860: Bootstrap starting.
I20260618 23:24:18.794685 9511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ae35dc6bac64997a96505c30e9e6860: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:18.794968 9511 log.cc:824] T 00000000000000000000000000000000 P 9ae35dc6bac64997a96505c30e9e6860: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:18.795655 9511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ae35dc6bac64997a96505c30e9e6860: No bootstrap required, opened a new log
I20260618 23:24:18.797678 9511 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ae35dc6bac64997a96505c30e9e6860 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae35dc6bac64997a96505c30e9e6860" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 34303 } }
I20260618 23:24:18.797832 9511 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ae35dc6bac64997a96505c30e9e6860 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:18.797878 9511 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ae35dc6bac64997a96505c30e9e6860 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ae35dc6bac64997a96505c30e9e6860, State: Initialized, Role: FOLLOWER
I20260618 23:24:18.798038 9511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ae35dc6bac64997a96505c30e9e6860 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae35dc6bac64997a96505c30e9e6860" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 34303 } }
I20260618 23:24:18.798132 9511 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ae35dc6bac64997a96505c30e9e6860 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:24:18.798177 9511 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ae35dc6bac64997a96505c30e9e6860 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:24:18.798224 9511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ae35dc6bac64997a96505c30e9e6860 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:18.798735 9511 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ae35dc6bac64997a96505c30e9e6860 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae35dc6bac64997a96505c30e9e6860" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 34303 } }
I20260618 23:24:18.798847 9511 leader_election.cc:302] T 00000000000000000000000000000000 P 9ae35dc6bac64997a96505c30e9e6860 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ae35dc6bac64997a96505c30e9e6860; no voters:
I20260618 23:24:18.799017 9511 leader_election.cc:288] T 00000000000000000000000000000000 P 9ae35dc6bac64997a96505c30e9e6860 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:24:18.799070 9514 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ae35dc6bac64997a96505c30e9e6860 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:18.799257 9511 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ae35dc6bac64997a96505c30e9e6860 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:24:18.799300 9514 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ae35dc6bac64997a96505c30e9e6860 [term 1 LEADER]: Becoming Leader. State: Replica: 9ae35dc6bac64997a96505c30e9e6860, State: Running, Role: LEADER
I20260618 23:24:18.799410 9514 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ae35dc6bac64997a96505c30e9e6860 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae35dc6bac64997a96505c30e9e6860" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 34303 } }
I20260618 23:24:18.799775 9515 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ae35dc6bac64997a96505c30e9e6860 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ae35dc6bac64997a96505c30e9e6860" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae35dc6bac64997a96505c30e9e6860" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 34303 } } }
I20260618 23:24:18.799880 9515 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ae35dc6bac64997a96505c30e9e6860 [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:18.799789 9516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ae35dc6bac64997a96505c30e9e6860 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ae35dc6bac64997a96505c30e9e6860. Latest consensus state: current_term: 1 leader_uuid: "9ae35dc6bac64997a96505c30e9e6860" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ae35dc6bac64997a96505c30e9e6860" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 34303 } } }
I20260618 23:24:18.800076 9516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ae35dc6bac64997a96505c30e9e6860 [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:18.800067 9523 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:24:18.800822 9523 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:24:18.801071 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 9433
I20260618 23:24:18.801192 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:24:18.801343 8350 external_mini_cluster.cc:1428] Setting key 129e1d6bb8e2dcfa328db7bed16dd465
I20260618 23:24:18.802475 9523 catalog_manager.cc:1347] Generated new cluster ID: d9a4300034f04b83abe32055d01a24ce
I20260618 23:24:18.802533 9523 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:24:18.809803 9523 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:24:18.810496 9523 catalog_manager.cc:1504] Loading token signing keys...
Jun 18 23:24:18 dist-test-slave-47qt krb5kdc[9420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:18.815927 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:18.801795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48636 (local address 127.8.39.190:34303)
0618 23:24:18.801951 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:18.801960 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:18.805352 (+ 3392us) server_negotiation.cc:407] Connection header received
0618 23:24:18.805593 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:18.805601 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:18.805659 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:18.805808 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:18.806538 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:18.808312 (+ 1774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:18.809413 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:18.810025 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:18.810159 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:18.812621 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:18.812626 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:18.812626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:18.812644 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:18.814365 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:18.814772 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:18.814774 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:18.814775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:18.814862 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:18.815074 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:18.815075 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:18.815075 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:18.815296 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:18.815439 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:18.815559 (+ 120us) server_negotiation.cc:299] Negotiation successful
0618 23:24:18.815710 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":82,"threads_started":1}
I20260618 23:24:18.823849 9523 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ae35dc6bac64997a96505c30e9e6860: Generated new TSK 0
I20260618 23:24:18.824049 9523 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.39.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
I20260618 23:24:18.855607 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.129:0
--local_ip_for_outbound_sockets=127.8.39.129
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:34303
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:44985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:18.956030 9537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:18.956260 9537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:18.956287 9537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:18.957906 9537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:18.957964 9537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:18.957988 9537 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 23:24:18.958009 9537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.129
I20260618 23:24:18.959717 9537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:44985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.39.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.39.190:34303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.129
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:18.959969 9537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:18.960219 9537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:18.965903 9543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:18.965909 9542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:18.965921 9545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:18.966216 9537 server_base.cc:1034] running on GCE node
I20260618 23:24:18.966503 9537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:18.966740 9537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:18.967929 9537 hybrid_clock.cc:648] HybridClock initialized: now 1781825058967908 us; error 28 us; skew 500 ppm
Jun 18 23:24:18 dist-test-slave-47qt krb5kdc[9420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825058, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:18.971072 9537 init.cc:377] Logged in from keytab as kudu/127.8.39.129@KRBTEST.COM (short username kudu)
I20260618 23:24:18.971576 9537 webserver.cc:460] Webserver started at http://127.8.39.129:35029/ using document root <none> and password file <none>
I20260618 23:24:18.971813 9537 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:18.971881 9537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:18.972016 9537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:18.973083 9537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-0/data/instance:
uuid: "19abf35f684045d2ad0d70ce826b3863"
format_stamp: "Formatted at 2026-06-18 23:24:18 on dist-test-slave-47qt"
server_key: "d795bfa4e7b2cab989037b5918538a6f"
server_key_iv: "e188b8203db6b338e5b49095dd925aeb"
server_key_version: "encryptionkey@0"
I20260618 23:24:18.973711 9537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance:
uuid: "19abf35f684045d2ad0d70ce826b3863"
format_stamp: "Formatted at 2026-06-18 23:24:18 on dist-test-slave-47qt"
server_key: "d795bfa4e7b2cab989037b5918538a6f"
server_key_iv: "e188b8203db6b338e5b49095dd925aeb"
server_key_version: "encryptionkey@0"
I20260618 23:24:18.975131 9537 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:18.975960 9552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:18.976215 9537 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:18.976321 9537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-0/wal
uuid: "19abf35f684045d2ad0d70ce826b3863"
format_stamp: "Formatted at 2026-06-18 23:24:18 on dist-test-slave-47qt"
server_key: "d795bfa4e7b2cab989037b5918538a6f"
server_key_iv: "e188b8203db6b338e5b49095dd925aeb"
server_key_version: "encryptionkey@0"
I20260618 23:24:18.976408 9537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:18.997073 9537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:19.000167 9537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:19.000335 9537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:19.001609 9537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:19.001679 9537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:19.001747 9537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:19.001781 9537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:19.009877 9537 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.129:38929
I20260618 23:24:19.009953 9667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.129:38929 every 8 connection(s)
I20260618 23:24:19.010279 9537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
I20260618 23:24:19.011255 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 9537
I20260618 23:24:19.011369 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance
I20260618 23:24:19.011534 8350 external_mini_cluster.cc:1428] Setting key fdbf958ecd98e093a32951733279a045
Jun 18 23:24:19 dist-test-slave-47qt krb5kdc[9420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825058, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:19.015874 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.001726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:43901 (local address 127.8.39.190:34303)
0618 23:24:19.002015 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:19.002017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:19.005433 (+ 3416us) server_negotiation.cc:407] Connection header received
0618 23:24:19.006316 (+ 883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:19.006317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:19.006334 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:19.006402 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:19.007363 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.007897 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.009725 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.010015 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.010096 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.012846 (+ 2750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:19.012847 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:19.012847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:19.012855 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:19.014418 (+ 1563us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:19.014917 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:19.014917 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:19.014918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:19.014968 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:19.015237 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:19.015238 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:19.015238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:19.015333 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:19.015382 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:19.015742 (+ 360us) server_negotiation.cc:299] Negotiation successful
0618 23:24:19.015812 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":267}
I20260618 23:24:19.016134 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.001757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:34303 (local address 127.8.39.129:43901)
0618 23:24:19.005255 (+ 3498us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:19.005313 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:19.006191 (+ 878us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:19.006500 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:19.006506 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:19.006917 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:19.007249 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.007255 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.008027 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.008029 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.009609 (+ 1580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.009613 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.010166 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.010167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.010268 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.010857 (+ 589us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:19.010866 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:19.012743 (+ 1877us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:19.014510 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:19.014515 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:19.014518 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:19.014835 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:19.015035 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:19.015036 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:19.015036 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:19.015174 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:19.015398 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:19.015404 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:19.015586 (+ 182us) client_negotiation.cc:769] Sending connection context
0618 23:24:19.015711 (+ 125us) client_negotiation.cc:240] Negotiation successful
0618 23:24:19.015891 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3447,"thread_start_us":97,"threads_started":1}
I20260618 23:24:19.016636 9668 heartbeater.cc:347] Connected to a master server at 127.8.39.190:34303
I20260618 23:24:19.016726 9668 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:19.016909 9668 heartbeater.cc:511] Master 127.8.39.190:34303 requested a full tablet report, sending...
I20260618 23:24:19.017479 9459 ts_manager.cc:194] Registered new tserver with Master: 19abf35f684045d2ad0d70ce826b3863 (127.8.39.129:38929)
I20260618 23:24:19.018090 9459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.129@KRBTEST.COM'} at 127.8.39.129:43901
WARNING: no policy specified for kudu/127.8.39.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.130@KRBTEST.COM" created.
I20260618 23:24:19.025460 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.018431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:53991 (local address 127.8.39.190:34303)
0618 23:24:19.018467 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:19.018469 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:19.018587 (+ 118us) server_negotiation.cc:407] Connection header received
0618 23:24:19.018657 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:19.018658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:19.018670 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:19.018743 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:19.019098 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.019878 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.020629 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.020816 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.020875 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.022189 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:19.022190 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:19.022190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:19.022196 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:19.024207 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:19.024604 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:19.024605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:19.024606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:19.024670 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:19.024873 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:19.024874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:19.024874 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:19.024984 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:19.025062 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:19.025289 (+ 227us) server_negotiation.cc:299] Negotiation successful
0618 23:24:19.025365 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260618 23:24:19.025476 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.018403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:34303 (local address 127.8.39.129:53991)
0618 23:24:19.018530 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:19.018540 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:19.018607 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:19.018748 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:19.018749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:19.018938 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:19.019012 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.019014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.019983 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.019983 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.020554 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.020557 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.020965 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.020967 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.021079 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.021802 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:19.021807 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:19.022102 (+ 295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:19.024327 (+ 2225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:19.024328 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:19.024329 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:19.024514 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:19.024742 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:19.024743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:19.024743 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:19.024804 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:19.025055 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:19.025056 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:19.025163 (+ 107us) client_negotiation.cc:769] Sending connection context
0618 23:24:19.025252 (+ 89us) client_negotiation.cc:240] Negotiation successful
0618 23:24:19.025367 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":68,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
I20260618 23:24:19.040386 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.130:0
--local_ip_for_outbound_sockets=127.8.39.130
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:34303
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:44985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:19.125851 9677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:19.126050 9677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:19.126077 9677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:19.127383 9677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:19.127418 9677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:19.127439 9677 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 23:24:19.127456 9677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.130
I20260618 23:24:19.129228 9677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:44985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.39.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.39.190:34303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.130
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:19.129546 9677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:19.129750 9677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:19.134713 9683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:19.134773 9682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:19.134763 9677 server_base.cc:1034] running on GCE node
W20260618 23:24:19.134723 9685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:19.135285 9677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:19.135581 9677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:19.136770 9677 hybrid_clock.cc:648] HybridClock initialized: now 1781825059136740 us; error 62 us; skew 500 ppm
Jun 18 23:24:19 dist-test-slave-47qt krb5kdc[9420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825059, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:19.139312 9677 init.cc:377] Logged in from keytab as kudu/127.8.39.130@KRBTEST.COM (short username kudu)
I20260618 23:24:19.139752 9677 webserver.cc:460] Webserver started at http://127.8.39.130:41177/ using document root <none> and password file <none>
I20260618 23:24:19.139925 9677 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:19.139966 9677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:19.140053 9677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:19.141031 9677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-1/data/instance:
uuid: "79a1d312cc5644d893085e5bde1e9d64"
format_stamp: "Formatted at 2026-06-18 23:24:19 on dist-test-slave-47qt"
server_key: "039762b8aec908bbb2935931d6451ea0"
server_key_iv: "38ac9758008946dc8bcb07cde5fd7227"
server_key_version: "encryptionkey@0"
I20260618 23:24:19.141394 9677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance:
uuid: "79a1d312cc5644d893085e5bde1e9d64"
format_stamp: "Formatted at 2026-06-18 23:24:19 on dist-test-slave-47qt"
server_key: "039762b8aec908bbb2935931d6451ea0"
server_key_iv: "38ac9758008946dc8bcb07cde5fd7227"
server_key_version: "encryptionkey@0"
I20260618 23:24:19.143013 9677 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:24:19.143668 9692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:19.143817 9677 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260618 23:24:19.143903 9677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-1/wal
uuid: "79a1d312cc5644d893085e5bde1e9d64"
format_stamp: "Formatted at 2026-06-18 23:24:19 on dist-test-slave-47qt"
server_key: "039762b8aec908bbb2935931d6451ea0"
server_key_iv: "38ac9758008946dc8bcb07cde5fd7227"
server_key_version: "encryptionkey@0"
I20260618 23:24:19.143958 9677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:19.161918 9677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:19.163787 9677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:19.163894 9677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:19.164337 9677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:19.164377 9677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:19.164400 9677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:19.164412 9677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:19.171448 9677 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.130:38953
I20260618 23:24:19.171522 9807 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.130:38953 every 8 connection(s)
I20260618 23:24:19.171799 9677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
Jun 18 23:24:19 dist-test-slave-47qt krb5kdc[9420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825059, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:19.175263 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 9677
I20260618 23:24:19.175324 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.164823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:53425 (local address 127.8.39.190:34303)
0618 23:24:19.164897 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:19.164900 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:19.165060 (+ 160us) server_negotiation.cc:407] Connection header received
0618 23:24:19.165649 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:19.165650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:19.165662 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:19.165715 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:19.166527 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.167328 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.168885 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.169236 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.169304 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.172564 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:19.172566 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:19.172566 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:19.172574 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:19.173935 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:19.174358 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:19.174359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:19.174360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:19.174417 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:19.174679 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:19.174680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:19.174680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:19.174798 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:19.174875 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:19.175148 (+ 273us) server_negotiation.cc:299] Negotiation successful
0618 23:24:19.175233 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260618 23:24:19.175374 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance
I20260618 23:24:19.175488 9701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.164799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:34303 (local address 127.8.39.130:53425)
0618 23:24:19.164944 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:19.164982 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:19.165500 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:19.165781 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:19.165785 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:19.166158 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:19.166440 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.166446 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.167409 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.167410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.168803 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.168808 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.169312 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.169313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.169450 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.170466 (+ 1016us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:19.170476 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:19.172466 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:19.174036 (+ 1570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:19.174043 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:19.174045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:19.174264 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:19.174486 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:19.174487 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:19.174487 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:19.174600 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:19.174866 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:19.174872 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:19.175008 (+ 136us) client_negotiation.cc:769] Sending connection context
0618 23:24:19.175116 (+ 108us) client_negotiation.cc:240] Negotiation successful
0618 23:24:19.175266 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260618 23:24:19.175559 8350 external_mini_cluster.cc:1428] Setting key 29bd489284e3229198b9731bfc6f348a
I20260618 23:24:19.175956 9808 heartbeater.cc:347] Connected to a master server at 127.8.39.190:34303
I20260618 23:24:19.176029 9808 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:19.176281 9808 heartbeater.cc:511] Master 127.8.39.190:34303 requested a full tablet report, sending...
I20260618 23:24:19.176651 9459 ts_manager.cc:194] Registered new tserver with Master: 79a1d312cc5644d893085e5bde1e9d64 (127.8.39.130:38953)
I20260618 23:24:19.177155 9459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.130@KRBTEST.COM'} at 127.8.39.130:53425
I20260618 23:24:19.183931 9814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.177016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:34303 (local address 127.8.39.130:55581)
0618 23:24:19.177164 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:19.177192 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:19.177270 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:19.177437 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:19.177438 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:19.177663 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:19.177735 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.177736 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.178411 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.178412 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.178983 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.178985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.179482 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.179484 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.179644 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.180305 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:19.180311 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:19.180984 (+ 673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:19.182867 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:19.182868 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:19.182868 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:19.183049 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:19.183261 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:19.183261 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:19.183262 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:19.183313 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:19.183677 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:19.183677 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:19.183717 (+ 40us) client_negotiation.cc:769] Sending connection context
0618 23:24:19.183770 (+ 53us) client_negotiation.cc:240] Negotiation successful
0618 23:24:19.183854 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
I20260618 23:24:19.183984 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.177075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:55581 (local address 127.8.39.190:34303)
0618 23:24:19.177110 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:19.177112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:19.177244 (+ 132us) server_negotiation.cc:407] Connection header received
0618 23:24:19.177338 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:19.177338 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:19.177349 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:19.177377 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:19.177820 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.178304 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.179069 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.179365 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.179438 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.181083 (+ 1645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:19.181085 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:19.181086 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:19.181093 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:19.182769 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:19.183127 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:19.183128 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:19.183129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:19.183192 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:19.183377 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:19.183378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:19.183378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:19.183592 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:19.183678 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:19.183816 (+ 138us) server_negotiation.cc:299] Negotiation successful
0618 23:24:19.183893 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
WARNING: no policy specified for kudu/127.8.39.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
I20260618 23:24:19.204095 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.131:0
--local_ip_for_outbound_sockets=127.8.39.131
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:34303
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:44985
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:19.287489 9817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:19.287714 9817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:19.287755 9817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:19.289460 9817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:19.289526 9817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:19.289734 9817 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 23:24:19.289778 9817 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.131
I20260618 23:24:19.291363 9817 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:44985
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.39.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.39.190:34303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.131
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:19.291852 9817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:19.292213 9817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:19.297159 9823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:19.297214 9825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:19.297338 9817 server_base.cc:1034] running on GCE node
W20260618 23:24:19.297385 9822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:19.297575 9817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:19.297818 9817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:19.298955 9817 hybrid_clock.cc:648] HybridClock initialized: now 1781825059298932 us; error 43 us; skew 500 ppm
Jun 18 23:24:19 dist-test-slave-47qt krb5kdc[9420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825059, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:19.302099 9817 init.cc:377] Logged in from keytab as kudu/127.8.39.131@KRBTEST.COM (short username kudu)
I20260618 23:24:19.302820 9817 webserver.cc:460] Webserver started at http://127.8.39.131:41815/ using document root <none> and password file <none>
I20260618 23:24:19.303064 9817 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:19.303110 9817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:19.303256 9817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:19.304279 9817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-2/data/instance:
uuid: "66d91f159ad2454f9979f9dc82b5a89b"
format_stamp: "Formatted at 2026-06-18 23:24:19 on dist-test-slave-47qt"
server_key: "e3204c2785f288caf93f11e0d7625307"
server_key_iv: "cfeb79ff773aba8b5fa1059c42f8d99e"
server_key_version: "encryptionkey@0"
I20260618 23:24:19.304634 9817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance:
uuid: "66d91f159ad2454f9979f9dc82b5a89b"
format_stamp: "Formatted at 2026-06-18 23:24:19 on dist-test-slave-47qt"
server_key: "e3204c2785f288caf93f11e0d7625307"
server_key_iv: "cfeb79ff773aba8b5fa1059c42f8d99e"
server_key_version: "encryptionkey@0"
I20260618 23:24:19.306104 9817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:24:19.307101 9832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:19.307323 9817 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:19.307426 9817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-2/wal
uuid: "66d91f159ad2454f9979f9dc82b5a89b"
format_stamp: "Formatted at 2026-06-18 23:24:19 on dist-test-slave-47qt"
server_key: "e3204c2785f288caf93f11e0d7625307"
server_key_iv: "cfeb79ff773aba8b5fa1059c42f8d99e"
server_key_version: "encryptionkey@0"
I20260618 23:24:19.307518 9817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:19.316859 9817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:19.318943 9817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:19.319089 9817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:19.319672 9817 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:19.319720 9817 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:19.319748 9817 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:19.319768 9817 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 23:24:19 dist-test-slave-47qt krb5kdc[9420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825059, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:19.326854 9817 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.131:44541
I20260618 23:24:19.326908 9947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.131:44541 every 8 connection(s)
I20260618 23:24:19.327198 9817 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
I20260618 23:24:19.329676 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 9817
I20260618 23:24:19.329754 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance
I20260618 23:24:19.329948 8350 external_mini_cluster.cc:1428] Setting key c90a660dafd8a2e0d3153bcafd48792d
I20260618 23:24:19.330644 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.319997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:40951 (local address 127.8.39.190:34303)
0618 23:24:19.320040 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:19.320042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:19.320256 (+ 214us) server_negotiation.cc:407] Connection header received
0618 23:24:19.320747 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:19.320748 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:19.320760 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:19.320815 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:19.321752 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.322541 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.324217 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.324545 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.324622 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.327347 (+ 2725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:19.327348 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:19.327348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:19.327356 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:19.328813 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:19.329343 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:19.329343 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:19.329344 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:19.329392 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:19.329861 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:19.329862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:19.329863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:19.330012 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:19.330087 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:19.330469 (+ 382us) server_negotiation.cc:299] Negotiation successful
0618 23:24:19.330551 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260618 23:24:19.330823 9841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.320002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:34303 (local address 127.8.39.131:40951)
0618 23:24:19.320148 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:19.320179 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:19.320648 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:19.320906 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:19.320910 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:19.321292 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:19.321616 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.321624 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.322643 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.322644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.324124 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.324127 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.324654 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.324655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.324826 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.325395 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:19.325406 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:19.327221 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:19.328938 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:19.328944 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:19.328946 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:19.329267 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:19.329612 (+ 345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:19.329613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:19.329613 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:19.329711 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:19.330122 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:19.330127 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:19.330295 (+ 168us) client_negotiation.cc:769] Sending connection context
0618 23:24:19.330402 (+ 107us) client_negotiation.cc:240] Negotiation successful
0618 23:24:19.330570 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":94,"threads_started":1}
I20260618 23:24:19.331364 9948 heartbeater.cc:347] Connected to a master server at 127.8.39.190:34303
I20260618 23:24:19.331466 9948 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:19.331693 9948 heartbeater.cc:511] Master 127.8.39.190:34303 requested a full tablet report, sending...
I20260618 23:24:19.332113 9465 ts_manager.cc:194] Registered new tserver with Master: 66d91f159ad2454f9979f9dc82b5a89b (127.8.39.131:44541)
I20260618 23:24:19.332537 9465 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.131@KRBTEST.COM'} at 127.8.39.131:40951
I20260618 23:24:19.337605 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.332626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:53515 (local address 127.8.39.190:34303)
0618 23:24:19.332676 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:19.332677 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:19.332856 (+ 179us) server_negotiation.cc:407] Connection header received
0618 23:24:19.332913 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:19.332914 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:19.332924 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:19.332981 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:19.333347 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.333786 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.334401 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.334588 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.334639 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.335446 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:19.335447 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:19.335447 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:19.335452 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:19.336683 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:19.336969 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:19.336969 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:19.336970 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:19.337007 (+ 37us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:19.337223 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:19.337224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:19.337224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:19.337307 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:19.337355 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:19.337493 (+ 138us) server_negotiation.cc:299] Negotiation successful
0618 23:24:19.337551 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260618 23:24:19.337616 9954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.332567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:34303 (local address 127.8.39.131:53515)
0618 23:24:19.332790 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:19.332799 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:19.332844 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:19.332987 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:19.332988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:19.333203 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:19.333278 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.333280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.333866 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.333866 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.334335 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.334337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.334669 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.334670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.334734 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.335089 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:19.335093 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:19.335375 (+ 282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:19.336757 (+ 1382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:19.336757 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:19.336758 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:19.336908 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:19.337059 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:19.337059 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:19.337059 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:19.337113 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:19.337361 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:19.337362 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:19.337396 (+ 34us) client_negotiation.cc:769] Sending connection context
0618 23:24:19.337477 (+ 81us) client_negotiation.cc:240] Negotiation successful
0618 23:24:19.337553 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":80,"threads_started":1}
I20260618 23:24:19.342271 8350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 18 23:24:19 dist-test-slave-47qt krb5kdc[9420](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825059, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 18 23:24:19 dist-test-slave-47qt krb5kdc[9420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825059, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:19.364616 9533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.357890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48652 (local address 127.8.39.190:34303)
0618 23:24:19.357958 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:19.357960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:19.357969 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:24:19.358049 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:19.358050 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:19.358058 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:19.358114 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:19.358655 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.359134 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.359840 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.360012 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.360066 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.362330 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:19.362331 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:19.362331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:19.362337 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:19.363725 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:19.364049 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:19.364049 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:19.364050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:19.364095 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:19.364257 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:19.364257 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:19.364257 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:19.364347 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:19.364419 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:19.364506 (+ 87us) server_negotiation.cc:299] Negotiation successful
0618 23:24:19.364557 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260618 23:24:19.366245 9465 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:24:19.367801 9465 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:24:19.374415 9965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.370237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:38929 (local address 127.0.0.1:38152)
0618 23:24:19.370516 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:19.370527 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:19.370587 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:19.370856 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:19.370859 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:19.370862 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:19.370929 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.370930 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.372026 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.372027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.373204 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.373207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.374099 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.374100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.374185 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.374199 (+ 14us) client_negotiation.cc:769] Sending connection context
0618 23:24:19.374278 (+ 79us) client_negotiation.cc:240] Negotiation successful
0618 23:24:19.374337 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":4,"thread_start_us":84,"threads_started":1}
I20260618 23:24:19.374444 9967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.370351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38152 (local address 127.8.39.129:38929)
0618 23:24:19.370538 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:19.370543 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:19.370575 (+ 32us) server_negotiation.cc:407] Connection header received
0618 23:24:19.370664 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:19.370666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:19.370681 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:19.370765 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:19.371022 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.371942 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.373285 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.374013 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.374088 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.374142 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:19.374301 (+ 159us) server_negotiation.cc:299] Negotiation successful
0618 23:24:19.374341 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":88,"threads_started":1}
I20260618 23:24:19.374861 9970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.370670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40658 (local address 127.8.39.130:38953)
0618 23:24:19.370862 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:19.370866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:19.371224 (+ 358us) server_negotiation.cc:407] Connection header received
0618 23:24:19.371265 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:19.371267 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:19.371282 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:19.371362 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:19.371466 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.372497 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.373715 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.374296 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.374354 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.374404 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:19.374717 (+ 313us) server_negotiation.cc:299] Negotiation successful
0618 23:24:19.374756 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":82,"threads_started":1}
I20260618 23:24:19.374866 9966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.370236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:44541 (local address 127.0.0.1:39466)
0618 23:24:19.370379 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:19.370422 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:19.370488 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:19.370992 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:19.370993 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:19.370995 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:19.371037 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.371038 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.372334 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.372335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.373618 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.373621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.374615 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.374616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.374673 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.374676 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:24:19.374751 (+ 75us) client_negotiation.cc:240] Negotiation successful
0618 23:24:19.374789 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"mutex_wait_us":24,"thread_start_us":43,"threads_started":1}
I20260618 23:24:19.374894 9969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.370390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39466 (local address 127.8.39.131:44541)
0618 23:24:19.370682 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:19.370693 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:19.370703 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:24:19.370809 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:19.370817 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:19.370841 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:19.370933 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:19.371130 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.372240 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.373714 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.374458 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.374529 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.374581 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:19.374765 (+ 184us) server_negotiation.cc:299] Negotiation successful
0618 23:24:19.374801 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":78,"threads_started":1}
I20260618 23:24:19.374662 9968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.370406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:38953 (local address 127.0.0.1:40658)
0618 23:24:19.371149 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:19.371158 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:19.371205 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:19.371372 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:19.371372 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:19.371374 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:19.371410 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.371411 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.372657 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.372658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.373623 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.373625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.374435 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.374436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.374503 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.374508 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:24:19.374560 (+ 52us) client_negotiation.cc:240] Negotiation successful
0618 23:24:19.374608 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":66,"threads_started":1}
I20260618 23:24:19.375460 9602 tablet_service.cc:1467] Processing CreateTablet for tablet 412a195b4e0f45cd820e663f732d9a5a (DEFAULT_TABLE table=test-table [id=e1faf61a00654356843e123eb31646fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:19.375614 9882 tablet_service.cc:1467] Processing CreateTablet for tablet 412a195b4e0f45cd820e663f732d9a5a (DEFAULT_TABLE table=test-table [id=e1faf61a00654356843e123eb31646fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:19.375864 9602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 412a195b4e0f45cd820e663f732d9a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:19.375975 9882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 412a195b4e0f45cd820e663f732d9a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:19.378538 9971 tablet_bootstrap.cc:492] T 412a195b4e0f45cd820e663f732d9a5a P 66d91f159ad2454f9979f9dc82b5a89b: Bootstrap starting.
I20260618 23:24:19.378809 9972 tablet_bootstrap.cc:492] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863: Bootstrap starting.
I20260618 23:24:19.379514 9971 tablet_bootstrap.cc:654] T 412a195b4e0f45cd820e663f732d9a5a P 66d91f159ad2454f9979f9dc82b5a89b: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:19.379514 9972 tablet_bootstrap.cc:654] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:19.379417 9742 tablet_service.cc:1467] Processing CreateTablet for tablet 412a195b4e0f45cd820e663f732d9a5a (DEFAULT_TABLE table=test-table [id=e1faf61a00654356843e123eb31646fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:19.379750 9742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 412a195b4e0f45cd820e663f732d9a5a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:19.379873 9972 log.cc:824] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:19.379873 9971 log.cc:824] T 412a195b4e0f45cd820e663f732d9a5a P 66d91f159ad2454f9979f9dc82b5a89b: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:19.380571 9972 tablet_bootstrap.cc:492] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863: No bootstrap required, opened a new log
I20260618 23:24:19.380656 9972 ts_tablet_manager.cc:1397] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:24:19.380669 9971 tablet_bootstrap.cc:492] T 412a195b4e0f45cd820e663f732d9a5a P 66d91f159ad2454f9979f9dc82b5a89b: No bootstrap required, opened a new log
I20260618 23:24:19.380726 9971 ts_tablet_manager.cc:1397] T 412a195b4e0f45cd820e663f732d9a5a P 66d91f159ad2454f9979f9dc82b5a89b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260618 23:24:19.382091 9972 raft_consensus.cc:348] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } }
I20260618 23:24:19.382241 9972 raft_consensus.cc:374] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:19.382284 9972 raft_consensus.cc:729] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19abf35f684045d2ad0d70ce826b3863, State: Initialized, Role: FOLLOWER
I20260618 23:24:19.382411 9972 consensus_queue.cc:260] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } }
I20260618 23:24:19.382656 9972 ts_tablet_manager.cc:1428] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260618 23:24:19.382678 9668 heartbeater.cc:503] Master 127.8.39.190:34303 was elected leader, sending a full tablet report...
I20260618 23:24:19.382827 9971 raft_consensus.cc:348] T 412a195b4e0f45cd820e663f732d9a5a P 66d91f159ad2454f9979f9dc82b5a89b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } }
I20260618 23:24:19.382992 9971 raft_consensus.cc:374] T 412a195b4e0f45cd820e663f732d9a5a P 66d91f159ad2454f9979f9dc82b5a89b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:19.383034 9971 raft_consensus.cc:729] T 412a195b4e0f45cd820e663f732d9a5a P 66d91f159ad2454f9979f9dc82b5a89b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66d91f159ad2454f9979f9dc82b5a89b, State: Initialized, Role: FOLLOWER
I20260618 23:24:19.383136 9971 consensus_queue.cc:260] T 412a195b4e0f45cd820e663f732d9a5a P 66d91f159ad2454f9979f9dc82b5a89b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } }
I20260618 23:24:19.383373 9971 ts_tablet_manager.cc:1428] T 412a195b4e0f45cd820e663f732d9a5a P 66d91f159ad2454f9979f9dc82b5a89b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:24:19.383375 9976 tablet_bootstrap.cc:492] T 412a195b4e0f45cd820e663f732d9a5a P 79a1d312cc5644d893085e5bde1e9d64: Bootstrap starting.
I20260618 23:24:19.383530 9948 heartbeater.cc:503] Master 127.8.39.190:34303 was elected leader, sending a full tablet report...
I20260618 23:24:19.384107 9976 tablet_bootstrap.cc:654] T 412a195b4e0f45cd820e663f732d9a5a P 79a1d312cc5644d893085e5bde1e9d64: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:19.384380 9976 log.cc:824] T 412a195b4e0f45cd820e663f732d9a5a P 79a1d312cc5644d893085e5bde1e9d64: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:19.385113 9976 tablet_bootstrap.cc:492] T 412a195b4e0f45cd820e663f732d9a5a P 79a1d312cc5644d893085e5bde1e9d64: No bootstrap required, opened a new log
I20260618 23:24:19.385196 9976 ts_tablet_manager.cc:1397] T 412a195b4e0f45cd820e663f732d9a5a P 79a1d312cc5644d893085e5bde1e9d64: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:24:19.386561 9976 raft_consensus.cc:348] T 412a195b4e0f45cd820e663f732d9a5a P 79a1d312cc5644d893085e5bde1e9d64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } }
I20260618 23:24:19.386675 9976 raft_consensus.cc:374] T 412a195b4e0f45cd820e663f732d9a5a P 79a1d312cc5644d893085e5bde1e9d64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:19.386695 9976 raft_consensus.cc:729] T 412a195b4e0f45cd820e663f732d9a5a P 79a1d312cc5644d893085e5bde1e9d64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79a1d312cc5644d893085e5bde1e9d64, State: Initialized, Role: FOLLOWER
I20260618 23:24:19.386776 9976 consensus_queue.cc:260] T 412a195b4e0f45cd820e663f732d9a5a P 79a1d312cc5644d893085e5bde1e9d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } }
I20260618 23:24:19.386950 9976 ts_tablet_manager.cc:1428] T 412a195b4e0f45cd820e663f732d9a5a P 79a1d312cc5644d893085e5bde1e9d64: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260618 23:24:19.387033 9808 heartbeater.cc:503] Master 127.8.39.190:34303 was elected leader, sending a full tablet report...
W20260618 23:24:19.422333 9809 tablet.cc:2369] T 412a195b4e0f45cd820e663f732d9a5a P 79a1d312cc5644d893085e5bde1e9d64: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 23:24:19.511309 9669 tablet.cc:2369] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:24:19.532943 9975 raft_consensus.cc:482] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:24:19.533049 9975 raft_consensus.cc:504] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } }
I20260618 23:24:19.533442 9975 leader_election.cc:288] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 66d91f159ad2454f9979f9dc82b5a89b (127.8.39.131:44541), 79a1d312cc5644d893085e5bde1e9d64 (127.8.39.130:38953)
I20260618 23:24:19.536448 9970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.533943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:59917 (local address 127.8.39.130:38953)
0618 23:24:19.533987 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:19.533989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:19.534006 (+ 17us) server_negotiation.cc:407] Connection header received
0618 23:24:19.534095 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:19.534095 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:19.534107 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:19.534177 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:19.534331 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.534774 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.535745 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.536095 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.536168 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.536176 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:19.536339 (+ 163us) server_negotiation.cc:299] Negotiation successful
0618 23:24:19.536383 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260618 23:24:19.536456 9981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.533834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:38953 (local address 127.8.39.129:59917)
0618 23:24:19.533923 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:19.533935 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:19.534010 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:19.534182 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:19.534183 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:19.534184 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:19.534264 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.534266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.534861 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.534862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.535662 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.535664 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.536189 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.536191 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.536261 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.536266 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:24:19.536317 (+ 51us) client_negotiation.cc:240] Negotiation successful
0618 23:24:19.536351 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"mutex_wait_us":34,"thread_start_us":51,"threads_started":1}
I20260618 23:24:19.536523 9969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.533740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:34341 (local address 127.8.39.131:44541)
0618 23:24:19.533808 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:19.533810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:19.533896 (+ 86us) server_negotiation.cc:407] Connection header received
0618 23:24:19.534001 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:19.534002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:19.534016 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:19.534070 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:19.534255 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.534776 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.535796 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.536127 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.536234 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.536242 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:19.536407 (+ 165us) server_negotiation.cc:299] Negotiation successful
0618 23:24:19.536446 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260618 23:24:19.536792 9980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.533651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:44541 (local address 127.8.39.129:34341)
0618 23:24:19.533810 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:19.533821 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:19.533915 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:19.534086 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:19.534087 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:19.534089 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:19.534175 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.534177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.534860 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.534861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.535675 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.535678 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.536251 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.536252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.536326 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.536331 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:24:19.536387 (+ 56us) client_negotiation.cc:240] Negotiation successful
0618 23:24:19.536426 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":53,"thread_start_us":86,"threads_started":1}
I20260618 23:24:19.536875 9762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412a195b4e0f45cd820e663f732d9a5a" candidate_uuid: "19abf35f684045d2ad0d70ce826b3863" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79a1d312cc5644d893085e5bde1e9d64" is_pre_election: true
I20260618 23:24:19.537014 9762 raft_consensus.cc:2454] T 412a195b4e0f45cd820e663f732d9a5a P 79a1d312cc5644d893085e5bde1e9d64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19abf35f684045d2ad0d70ce826b3863 in term 0.
I20260618 23:24:19.537101 9902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412a195b4e0f45cd820e663f732d9a5a" candidate_uuid: "19abf35f684045d2ad0d70ce826b3863" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66d91f159ad2454f9979f9dc82b5a89b" is_pre_election: true
I20260618 23:24:19.537240 9553 leader_election.cc:302] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19abf35f684045d2ad0d70ce826b3863, 79a1d312cc5644d893085e5bde1e9d64; no voters:
I20260618 23:24:19.537264 9902 raft_consensus.cc:2454] T 412a195b4e0f45cd820e663f732d9a5a P 66d91f159ad2454f9979f9dc82b5a89b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19abf35f684045d2ad0d70ce826b3863 in term 0.
I20260618 23:24:19.537370 9975 raft_consensus.cc:2781] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:24:19.537423 9975 raft_consensus.cc:482] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:24:19.537446 9975 raft_consensus.cc:3037] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:19.538100 9975 raft_consensus.cc:504] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } }
I20260618 23:24:19.538231 9975 leader_election.cc:288] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863 [CANDIDATE]: Term 1 election: Requested vote from peers 66d91f159ad2454f9979f9dc82b5a89b (127.8.39.131:44541), 79a1d312cc5644d893085e5bde1e9d64 (127.8.39.130:38953)
I20260618 23:24:19.538412 9762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412a195b4e0f45cd820e663f732d9a5a" candidate_uuid: "19abf35f684045d2ad0d70ce826b3863" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79a1d312cc5644d893085e5bde1e9d64"
I20260618 23:24:19.538444 9902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "412a195b4e0f45cd820e663f732d9a5a" candidate_uuid: "19abf35f684045d2ad0d70ce826b3863" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66d91f159ad2454f9979f9dc82b5a89b"
I20260618 23:24:19.538509 9762 raft_consensus.cc:3037] T 412a195b4e0f45cd820e663f732d9a5a P 79a1d312cc5644d893085e5bde1e9d64 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:19.538504 9902 raft_consensus.cc:3037] T 412a195b4e0f45cd820e663f732d9a5a P 66d91f159ad2454f9979f9dc82b5a89b [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:19.539152 9762 raft_consensus.cc:2454] T 412a195b4e0f45cd820e663f732d9a5a P 79a1d312cc5644d893085e5bde1e9d64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19abf35f684045d2ad0d70ce826b3863 in term 1.
I20260618 23:24:19.539297 9902 raft_consensus.cc:2454] T 412a195b4e0f45cd820e663f732d9a5a P 66d91f159ad2454f9979f9dc82b5a89b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19abf35f684045d2ad0d70ce826b3863 in term 1.
I20260618 23:24:19.539343 9553 leader_election.cc:302] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19abf35f684045d2ad0d70ce826b3863, 79a1d312cc5644d893085e5bde1e9d64; no voters:
I20260618 23:24:19.539474 9975 raft_consensus.cc:2781] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:19.539588 9975 raft_consensus.cc:686] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863 [term 1 LEADER]: Becoming Leader. State: Replica: 19abf35f684045d2ad0d70ce826b3863, State: Running, Role: LEADER
I20260618 23:24:19.539678 9975 consensus_queue.cc:237] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } }
I20260618 23:24:19.540416 9459 catalog_manager.cc:5495] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19abf35f684045d2ad0d70ce826b3863 (127.8.39.129). New cstate: current_term: 1 leader_uuid: "19abf35f684045d2ad0d70ce826b3863" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } health_report { overall_health: HEALTHY } } }
I20260618 23:24:19.563083 9967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.559759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38156 (local address 127.8.39.129:38929)
0618 23:24:19.559817 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:19.559819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:19.559829 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:24:19.559927 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:19.559928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:19.559958 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:19.560026 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:24:19.560189 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.560683 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.561936 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.562222 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.562300 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.562577 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:24:19.562712 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:24:19.562811 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:19.562950 (+ 139us) server_negotiation.cc:299] Negotiation successful
0618 23:24:19.562982 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260618 23:24:19.565779 9762 raft_consensus.cc:1261] T 412a195b4e0f45cd820e663f732d9a5a P 79a1d312cc5644d893085e5bde1e9d64 [term 1 FOLLOWER]: Refusing update from remote peer 19abf35f684045d2ad0d70ce826b3863: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:24:19.565841 9902 raft_consensus.cc:1261] T 412a195b4e0f45cd820e663f732d9a5a P 66d91f159ad2454f9979f9dc82b5a89b [term 1 FOLLOWER]: Refusing update from remote peer 19abf35f684045d2ad0d70ce826b3863: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:24:19.566182 9975 consensus_queue.cc:1035] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:19.566337 9982 consensus_queue.cc:1035] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:19.569165 9985 mvcc.cc:205] Tried to move back new op lower bound from 7298355443977445376 to 7298355443875258368. Current Snapshot: MvccSnapshot[applied={T|T < 7298355443977445376}]
I20260618 23:24:19.569799 9987 mvcc.cc:205] Tried to move back new op lower bound from 7298355443977445376 to 7298355443875258368. Current Snapshot: MvccSnapshot[applied={T|T < 7298355443977445376}]
I20260618 23:24:19.569845 9986 mvcc.cc:205] Tried to move back new op lower bound from 7298355443977445376 to 7298355443875258368. Current Snapshot: MvccSnapshot[applied={T|T < 7298355443977445376}]
I20260618 23:24:19.573513 9459 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:24:19.573616 9459 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:24:19.574450 9459 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ae35dc6bac64997a96505c30e9e6860: Sending DeleteTablet for 3 replicas of tablet 412a195b4e0f45cd820e663f732d9a5a
I20260618 23:24:19.574848 9882 tablet_service.cc:1514] Processing DeleteTablet for tablet 412a195b4e0f45cd820e663f732d9a5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:19 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:39466
I20260618 23:24:19.574901 9602 tablet_service.cc:1514] Processing DeleteTablet for tablet 412a195b4e0f45cd820e663f732d9a5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:19 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:38152
I20260618 23:24:19.574882 9742 tablet_service.cc:1514] Processing DeleteTablet for tablet 412a195b4e0f45cd820e663f732d9a5a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:19 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:40658
I20260618 23:24:19.575084 9995 tablet_replica.cc:330] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863: stopping tablet replica
I20260618 23:24:19.575213 9995 raft_consensus.cc:2229] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863 [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:24:19.575309 9994 tablet_replica.cc:330] T 412a195b4e0f45cd820e663f732d9a5a P 66d91f159ad2454f9979f9dc82b5a89b: stopping tablet replica
I20260618 23:24:19.575440 9994 raft_consensus.cc:2229] T 412a195b4e0f45cd820e663f732d9a5a P 66d91f159ad2454f9979f9dc82b5a89b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:24:19.575389 9459 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:24:19.575542 9459 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:24:19.575559 9994 raft_consensus.cc:2258] T 412a195b4e0f45cd820e663f732d9a5a P 66d91f159ad2454f9979f9dc82b5a89b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:19.576187 9994 ts_tablet_manager.cc:1905] T 412a195b4e0f45cd820e663f732d9a5a P 66d91f159ad2454f9979f9dc82b5a89b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:19.576404 9995 raft_consensus.cc:2258] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:19.576610 9996 tablet_replica.cc:330] T 412a195b4e0f45cd820e663f732d9a5a P 79a1d312cc5644d893085e5bde1e9d64: stopping tablet replica
I20260618 23:24:19.576711 9996 raft_consensus.cc:2229] T 412a195b4e0f45cd820e663f732d9a5a P 79a1d312cc5644d893085e5bde1e9d64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:24:19.576752 9602 tablet_service.cc:1467] Processing CreateTablet for tablet f99e26e014184aa0845b1ac16bf638a7 (DEFAULT_TABLE table=test-table [id=792ca5cff30f405f9cef5cd5f61fec63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:19.576846 9996 raft_consensus.cc:2258] T 412a195b4e0f45cd820e663f732d9a5a P 79a1d312cc5644d893085e5bde1e9d64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:19.576861 9602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f99e26e014184aa0845b1ac16bf638a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:19.577270 9742 tablet_service.cc:1467] Processing CreateTablet for tablet f99e26e014184aa0845b1ac16bf638a7 (DEFAULT_TABLE table=test-table [id=792ca5cff30f405f9cef5cd5f61fec63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:19.577373 9742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f99e26e014184aa0845b1ac16bf638a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:19.577987 9972 tablet_bootstrap.cc:492] T f99e26e014184aa0845b1ac16bf638a7 P 19abf35f684045d2ad0d70ce826b3863: Bootstrap starting.
I20260618 23:24:19.578444 9972 tablet_bootstrap.cc:654] T f99e26e014184aa0845b1ac16bf638a7 P 19abf35f684045d2ad0d70ce826b3863: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:19.578697 9976 tablet_bootstrap.cc:492] T f99e26e014184aa0845b1ac16bf638a7 P 79a1d312cc5644d893085e5bde1e9d64: Bootstrap starting.
I20260618 23:24:19.578948 9996 ts_tablet_manager.cc:1905] T 412a195b4e0f45cd820e663f732d9a5a P 79a1d312cc5644d893085e5bde1e9d64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:19.579118 9976 tablet_bootstrap.cc:654] T f99e26e014184aa0845b1ac16bf638a7 P 79a1d312cc5644d893085e5bde1e9d64: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:19.577107 9882 tablet_service.cc:1467] Processing CreateTablet for tablet f99e26e014184aa0845b1ac16bf638a7 (DEFAULT_TABLE table=test-table [id=792ca5cff30f405f9cef5cd5f61fec63]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:19.579638 9882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f99e26e014184aa0845b1ac16bf638a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:19.580646 9996 ts_tablet_manager.cc:1918] T 412a195b4e0f45cd820e663f732d9a5a P 79a1d312cc5644d893085e5bde1e9d64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:24:19.580703 9995 ts_tablet_manager.cc:1905] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:19.580735 9996 log.cc:1196] T 412a195b4e0f45cd820e663f732d9a5a P 79a1d312cc5644d893085e5bde1e9d64: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-1/wal/wals/412a195b4e0f45cd820e663f732d9a5a
I20260618 23:24:19.581015 9996 ts_tablet_manager.cc:1939] T 412a195b4e0f45cd820e663f732d9a5a P 79a1d312cc5644d893085e5bde1e9d64: Deleting consensus metadata
I20260618 23:24:19.581027 9971 tablet_bootstrap.cc:492] T f99e26e014184aa0845b1ac16bf638a7 P 66d91f159ad2454f9979f9dc82b5a89b: Bootstrap starting.
I20260618 23:24:19.581295 9976 tablet_bootstrap.cc:492] T f99e26e014184aa0845b1ac16bf638a7 P 79a1d312cc5644d893085e5bde1e9d64: No bootstrap required, opened a new log
I20260618 23:24:19.581337 9976 ts_tablet_manager.cc:1397] T f99e26e014184aa0845b1ac16bf638a7 P 79a1d312cc5644d893085e5bde1e9d64: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260618 23:24:19.581454 9971 tablet_bootstrap.cc:654] T f99e26e014184aa0845b1ac16bf638a7 P 66d91f159ad2454f9979f9dc82b5a89b: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:19.581480 9976 raft_consensus.cc:348] T f99e26e014184aa0845b1ac16bf638a7 P 79a1d312cc5644d893085e5bde1e9d64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } }
I20260618 23:24:19.581562 9976 raft_consensus.cc:374] T f99e26e014184aa0845b1ac16bf638a7 P 79a1d312cc5644d893085e5bde1e9d64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:19.581585 9976 raft_consensus.cc:729] T f99e26e014184aa0845b1ac16bf638a7 P 79a1d312cc5644d893085e5bde1e9d64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79a1d312cc5644d893085e5bde1e9d64, State: Initialized, Role: FOLLOWER
I20260618 23:24:19.581636 9976 consensus_queue.cc:260] T f99e26e014184aa0845b1ac16bf638a7 P 79a1d312cc5644d893085e5bde1e9d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } }
I20260618 23:24:19.581687 9972 tablet_bootstrap.cc:492] T f99e26e014184aa0845b1ac16bf638a7 P 19abf35f684045d2ad0d70ce826b3863: No bootstrap required, opened a new log
I20260618 23:24:19.581724 9976 ts_tablet_manager.cc:1428] T f99e26e014184aa0845b1ac16bf638a7 P 79a1d312cc5644d893085e5bde1e9d64: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:19.581732 9972 ts_tablet_manager.cc:1397] T f99e26e014184aa0845b1ac16bf638a7 P 19abf35f684045d2ad0d70ce826b3863: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260618 23:24:19.581871 9972 raft_consensus.cc:348] T f99e26e014184aa0845b1ac16bf638a7 P 19abf35f684045d2ad0d70ce826b3863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } }
I20260618 23:24:19.581938 9972 raft_consensus.cc:374] T f99e26e014184aa0845b1ac16bf638a7 P 19abf35f684045d2ad0d70ce826b3863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:19.581959 9972 raft_consensus.cc:729] T f99e26e014184aa0845b1ac16bf638a7 P 19abf35f684045d2ad0d70ce826b3863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19abf35f684045d2ad0d70ce826b3863, State: Initialized, Role: FOLLOWER
I20260618 23:24:19.582000 9972 consensus_queue.cc:260] T f99e26e014184aa0845b1ac16bf638a7 P 19abf35f684045d2ad0d70ce826b3863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } }
I20260618 23:24:19.582015 9449 catalog_manager.cc:4841] TS 79a1d312cc5644d893085e5bde1e9d64 (127.8.39.130:38953): tablet 412a195b4e0f45cd820e663f732d9a5a (table test-table [id=e1faf61a00654356843e123eb31646fd]) successfully deleted
I20260618 23:24:19.582069 9972 ts_tablet_manager.cc:1428] T f99e26e014184aa0845b1ac16bf638a7 P 19abf35f684045d2ad0d70ce826b3863: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260618 23:24:19.582317 9995 ts_tablet_manager.cc:1918] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:24:19.582387 9995 log.cc:1196] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-0/wal/wals/412a195b4e0f45cd820e663f732d9a5a
I20260618 23:24:19.582647 9995 ts_tablet_manager.cc:1939] T 412a195b4e0f45cd820e663f732d9a5a P 19abf35f684045d2ad0d70ce826b3863: Deleting consensus metadata
I20260618 23:24:19.583041 9451 catalog_manager.cc:4841] TS 19abf35f684045d2ad0d70ce826b3863 (127.8.39.129:38929): tablet 412a195b4e0f45cd820e663f732d9a5a (table test-table [id=e1faf61a00654356843e123eb31646fd]) successfully deleted
I20260618 23:24:19.584230 9994 ts_tablet_manager.cc:1918] T 412a195b4e0f45cd820e663f732d9a5a P 66d91f159ad2454f9979f9dc82b5a89b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:24:19.584306 9994 log.cc:1196] T 412a195b4e0f45cd820e663f732d9a5a P 66d91f159ad2454f9979f9dc82b5a89b: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-2/wal/wals/412a195b4e0f45cd820e663f732d9a5a
I20260618 23:24:19.584558 9994 ts_tablet_manager.cc:1939] T 412a195b4e0f45cd820e663f732d9a5a P 66d91f159ad2454f9979f9dc82b5a89b: Deleting consensus metadata
I20260618 23:24:19.584972 9449 catalog_manager.cc:4841] TS 66d91f159ad2454f9979f9dc82b5a89b (127.8.39.131:44541): tablet 412a195b4e0f45cd820e663f732d9a5a (table test-table [id=e1faf61a00654356843e123eb31646fd]) successfully deleted
I20260618 23:24:19.585282 9971 tablet_bootstrap.cc:492] T f99e26e014184aa0845b1ac16bf638a7 P 66d91f159ad2454f9979f9dc82b5a89b: No bootstrap required, opened a new log
I20260618 23:24:19.585345 9971 ts_tablet_manager.cc:1397] T f99e26e014184aa0845b1ac16bf638a7 P 66d91f159ad2454f9979f9dc82b5a89b: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260618 23:24:19.585484 9971 raft_consensus.cc:348] T f99e26e014184aa0845b1ac16bf638a7 P 66d91f159ad2454f9979f9dc82b5a89b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } }
I20260618 23:24:19.585572 9971 raft_consensus.cc:374] T f99e26e014184aa0845b1ac16bf638a7 P 66d91f159ad2454f9979f9dc82b5a89b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:19.585593 9971 raft_consensus.cc:729] T f99e26e014184aa0845b1ac16bf638a7 P 66d91f159ad2454f9979f9dc82b5a89b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66d91f159ad2454f9979f9dc82b5a89b, State: Initialized, Role: FOLLOWER
I20260618 23:24:19.585633 9971 consensus_queue.cc:260] T f99e26e014184aa0845b1ac16bf638a7 P 66d91f159ad2454f9979f9dc82b5a89b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } }
I20260618 23:24:19.585726 9971 ts_tablet_manager.cc:1428] T f99e26e014184aa0845b1ac16bf638a7 P 66d91f159ad2454f9979f9dc82b5a89b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260618 23:24:19.828974 9949 tablet.cc:2369] T f99e26e014184aa0845b1ac16bf638a7 P 66d91f159ad2454f9979f9dc82b5a89b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:24:19.846028 9975 raft_consensus.cc:482] T f99e26e014184aa0845b1ac16bf638a7 P 19abf35f684045d2ad0d70ce826b3863 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:24:19.846125 9975 raft_consensus.cc:504] T f99e26e014184aa0845b1ac16bf638a7 P 19abf35f684045d2ad0d70ce826b3863 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } }
I20260618 23:24:19.846410 9975 leader_election.cc:288] T f99e26e014184aa0845b1ac16bf638a7 P 19abf35f684045d2ad0d70ce826b3863 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79a1d312cc5644d893085e5bde1e9d64 (127.8.39.130:38953), 66d91f159ad2454f9979f9dc82b5a89b (127.8.39.131:44541)
I20260618 23:24:19.846715 9762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f99e26e014184aa0845b1ac16bf638a7" candidate_uuid: "19abf35f684045d2ad0d70ce826b3863" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79a1d312cc5644d893085e5bde1e9d64" is_pre_election: true
I20260618 23:24:19.846848 9762 raft_consensus.cc:2454] T f99e26e014184aa0845b1ac16bf638a7 P 79a1d312cc5644d893085e5bde1e9d64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19abf35f684045d2ad0d70ce826b3863 in term 0.
I20260618 23:24:19.846844 9902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f99e26e014184aa0845b1ac16bf638a7" candidate_uuid: "19abf35f684045d2ad0d70ce826b3863" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66d91f159ad2454f9979f9dc82b5a89b" is_pre_election: true
I20260618 23:24:19.846966 9902 raft_consensus.cc:2454] T f99e26e014184aa0845b1ac16bf638a7 P 66d91f159ad2454f9979f9dc82b5a89b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 19abf35f684045d2ad0d70ce826b3863 in term 0.
I20260618 23:24:19.847047 9553 leader_election.cc:302] T f99e26e014184aa0845b1ac16bf638a7 P 19abf35f684045d2ad0d70ce826b3863 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19abf35f684045d2ad0d70ce826b3863, 79a1d312cc5644d893085e5bde1e9d64; no voters:
I20260618 23:24:19.847173 9975 raft_consensus.cc:2781] T f99e26e014184aa0845b1ac16bf638a7 P 19abf35f684045d2ad0d70ce826b3863 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:24:19.847225 9975 raft_consensus.cc:482] T f99e26e014184aa0845b1ac16bf638a7 P 19abf35f684045d2ad0d70ce826b3863 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:24:19.847240 9975 raft_consensus.cc:3037] T f99e26e014184aa0845b1ac16bf638a7 P 19abf35f684045d2ad0d70ce826b3863 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:19.847983 9975 raft_consensus.cc:504] T f99e26e014184aa0845b1ac16bf638a7 P 19abf35f684045d2ad0d70ce826b3863 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } }
I20260618 23:24:19.848076 9975 leader_election.cc:288] T f99e26e014184aa0845b1ac16bf638a7 P 19abf35f684045d2ad0d70ce826b3863 [CANDIDATE]: Term 1 election: Requested vote from peers 79a1d312cc5644d893085e5bde1e9d64 (127.8.39.130:38953), 66d91f159ad2454f9979f9dc82b5a89b (127.8.39.131:44541)
I20260618 23:24:19.848300 9902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f99e26e014184aa0845b1ac16bf638a7" candidate_uuid: "19abf35f684045d2ad0d70ce826b3863" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "66d91f159ad2454f9979f9dc82b5a89b"
I20260618 23:24:19.848359 9902 raft_consensus.cc:3037] T f99e26e014184aa0845b1ac16bf638a7 P 66d91f159ad2454f9979f9dc82b5a89b [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:19.848304 9762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f99e26e014184aa0845b1ac16bf638a7" candidate_uuid: "19abf35f684045d2ad0d70ce826b3863" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79a1d312cc5644d893085e5bde1e9d64"
I20260618 23:24:19.848402 9762 raft_consensus.cc:3037] T f99e26e014184aa0845b1ac16bf638a7 P 79a1d312cc5644d893085e5bde1e9d64 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:19.848939 9902 raft_consensus.cc:2454] T f99e26e014184aa0845b1ac16bf638a7 P 66d91f159ad2454f9979f9dc82b5a89b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19abf35f684045d2ad0d70ce826b3863 in term 1.
I20260618 23:24:19.849107 9762 raft_consensus.cc:2454] T f99e26e014184aa0845b1ac16bf638a7 P 79a1d312cc5644d893085e5bde1e9d64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19abf35f684045d2ad0d70ce826b3863 in term 1.
I20260618 23:24:19.849105 9553 leader_election.cc:302] T f99e26e014184aa0845b1ac16bf638a7 P 19abf35f684045d2ad0d70ce826b3863 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19abf35f684045d2ad0d70ce826b3863, 66d91f159ad2454f9979f9dc82b5a89b; no voters:
I20260618 23:24:19.849260 9975 raft_consensus.cc:2781] T f99e26e014184aa0845b1ac16bf638a7 P 19abf35f684045d2ad0d70ce826b3863 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:19.849310 9975 raft_consensus.cc:686] T f99e26e014184aa0845b1ac16bf638a7 P 19abf35f684045d2ad0d70ce826b3863 [term 1 LEADER]: Becoming Leader. State: Replica: 19abf35f684045d2ad0d70ce826b3863, State: Running, Role: LEADER
I20260618 23:24:19.849362 9975 consensus_queue.cc:237] T f99e26e014184aa0845b1ac16bf638a7 P 19abf35f684045d2ad0d70ce826b3863 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } }
I20260618 23:24:19.849934 9459 catalog_manager.cc:5495] T f99e26e014184aa0845b1ac16bf638a7 P 19abf35f684045d2ad0d70ce826b3863 reported cstate change: term changed from 0 to 1, leader changed from <none> to 19abf35f684045d2ad0d70ce826b3863 (127.8.39.129). New cstate: current_term: 1 leader_uuid: "19abf35f684045d2ad0d70ce826b3863" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } health_report { overall_health: UNKNOWN } } }
Jun 18 23:24:19 dist-test-slave-47qt krb5kdc[9420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825058, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:19.901476 10006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.893793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:34303 (local address 127.0.0.1:48654)
0618 23:24:19.893887 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:19.893899 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:19.893972 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:19.894134 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:19.894134 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:19.894519 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:19.894566 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.894569 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.895215 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.895216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.895742 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.895743 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.896122 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.896123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.896165 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.896756 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:19.896764 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:19.898684 (+ 1920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:19.900378 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:19.900385 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:19.900387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:19.900661 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:19.900881 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:19.900882 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:19.900882 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:19.900936 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:19.901220 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:19.901225 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:19.901265 (+ 40us) client_negotiation.cc:769] Sending connection context
0618 23:24:19.901318 (+ 53us) client_negotiation.cc:240] Negotiation successful
0618 23:24:19.901401 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"thread_start_us":40,"threads_started":1}
I20260618 23:24:19.901652 10007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.893857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48654 (local address 127.8.39.190:34303)
0618 23:24:19.894005 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:19.894006 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:19.894016 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:24:19.894038 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:19.894038 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:19.894049 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:19.894116 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:19.894637 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.895127 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.895837 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.896037 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.896115 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.898767 (+ 2652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:19.898768 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:19.898768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:19.898775 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:19.900310 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:19.900755 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:19.900756 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:19.900757 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:19.900814 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:19.900991 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:19.900991 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:19.900991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:19.901166 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:19.901218 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:19.901386 (+ 168us) server_negotiation.cc:299] Negotiation successful
0618 23:24:19.901468 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":81,"threads_started":1}
I20260618 23:24:19.903756 9459 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:48654:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260618 23:24:19.903932 9459 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:24:19.905447 9602 tablet_service.cc:1467] Processing CreateTablet for tablet bc5f5036f46941a4bd6b54254a78c13a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=41b6ccc98cc244e6a8b3dbebfada8fa0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 23:24:19.905483 9882 tablet_service.cc:1467] Processing CreateTablet for tablet bc5f5036f46941a4bd6b54254a78c13a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=41b6ccc98cc244e6a8b3dbebfada8fa0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 23:24:19.905484 9742 tablet_service.cc:1467] Processing CreateTablet for tablet bc5f5036f46941a4bd6b54254a78c13a (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=41b6ccc98cc244e6a8b3dbebfada8fa0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 23:24:19.905632 9602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc5f5036f46941a4bd6b54254a78c13a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:19.905632 9882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc5f5036f46941a4bd6b54254a78c13a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:19.905633 9742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bc5f5036f46941a4bd6b54254a78c13a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:19.906698 9976 tablet_bootstrap.cc:492] T bc5f5036f46941a4bd6b54254a78c13a P 79a1d312cc5644d893085e5bde1e9d64: Bootstrap starting.
I20260618 23:24:19.907051 9971 tablet_bootstrap.cc:492] T bc5f5036f46941a4bd6b54254a78c13a P 66d91f159ad2454f9979f9dc82b5a89b: Bootstrap starting.
I20260618 23:24:19.907096 9972 tablet_bootstrap.cc:492] T bc5f5036f46941a4bd6b54254a78c13a P 19abf35f684045d2ad0d70ce826b3863: Bootstrap starting.
I20260618 23:24:19.907207 9976 tablet_bootstrap.cc:654] T bc5f5036f46941a4bd6b54254a78c13a P 79a1d312cc5644d893085e5bde1e9d64: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:19.907416 9972 tablet_bootstrap.cc:654] T bc5f5036f46941a4bd6b54254a78c13a P 19abf35f684045d2ad0d70ce826b3863: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:19.907455 9971 tablet_bootstrap.cc:654] T bc5f5036f46941a4bd6b54254a78c13a P 66d91f159ad2454f9979f9dc82b5a89b: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:19.907876 9976 tablet_bootstrap.cc:492] T bc5f5036f46941a4bd6b54254a78c13a P 79a1d312cc5644d893085e5bde1e9d64: No bootstrap required, opened a new log
I20260618 23:24:19.907949 9976 ts_tablet_manager.cc:1397] T bc5f5036f46941a4bd6b54254a78c13a P 79a1d312cc5644d893085e5bde1e9d64: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260618 23:24:19.908034 9971 tablet_bootstrap.cc:492] T bc5f5036f46941a4bd6b54254a78c13a P 66d91f159ad2454f9979f9dc82b5a89b: No bootstrap required, opened a new log
I20260618 23:24:19.908090 9971 ts_tablet_manager.cc:1397] T bc5f5036f46941a4bd6b54254a78c13a P 66d91f159ad2454f9979f9dc82b5a89b: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260618 23:24:19.908128 9976 raft_consensus.cc:348] T bc5f5036f46941a4bd6b54254a78c13a P 79a1d312cc5644d893085e5bde1e9d64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } }
I20260618 23:24:19.908203 9976 raft_consensus.cc:374] T bc5f5036f46941a4bd6b54254a78c13a P 79a1d312cc5644d893085e5bde1e9d64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:19.908233 9976 raft_consensus.cc:729] T bc5f5036f46941a4bd6b54254a78c13a P 79a1d312cc5644d893085e5bde1e9d64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79a1d312cc5644d893085e5bde1e9d64, State: Initialized, Role: FOLLOWER
I20260618 23:24:19.908257 9971 raft_consensus.cc:348] T bc5f5036f46941a4bd6b54254a78c13a P 66d91f159ad2454f9979f9dc82b5a89b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } }
I20260618 23:24:19.908282 9976 consensus_queue.cc:260] T bc5f5036f46941a4bd6b54254a78c13a P 79a1d312cc5644d893085e5bde1e9d64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } }
I20260618 23:24:19.908326 9971 raft_consensus.cc:374] T bc5f5036f46941a4bd6b54254a78c13a P 66d91f159ad2454f9979f9dc82b5a89b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:19.908357 9971 raft_consensus.cc:729] T bc5f5036f46941a4bd6b54254a78c13a P 66d91f159ad2454f9979f9dc82b5a89b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66d91f159ad2454f9979f9dc82b5a89b, State: Initialized, Role: FOLLOWER
I20260618 23:24:19.908378 9976 ts_tablet_manager.cc:1428] T bc5f5036f46941a4bd6b54254a78c13a P 79a1d312cc5644d893085e5bde1e9d64: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260618 23:24:19.908401 9971 consensus_queue.cc:260] T bc5f5036f46941a4bd6b54254a78c13a P 66d91f159ad2454f9979f9dc82b5a89b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } }
I20260618 23:24:19.908416 9979 tablet_replica.cc:439] T bc5f5036f46941a4bd6b54254a78c13a P 79a1d312cc5644d893085e5bde1e9d64: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } } }
I20260618 23:24:19.908495 9971 ts_tablet_manager.cc:1428] T bc5f5036f46941a4bd6b54254a78c13a P 66d91f159ad2454f9979f9dc82b5a89b: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260618 23:24:19.908516 9979 tablet_replica.cc:442] T bc5f5036f46941a4bd6b54254a78c13a P 79a1d312cc5644d893085e5bde1e9d64: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:24:19.908613 9977 tablet_replica.cc:439] T bc5f5036f46941a4bd6b54254a78c13a P 66d91f159ad2454f9979f9dc82b5a89b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } } }
I20260618 23:24:19.908672 9977 tablet_replica.cc:442] T bc5f5036f46941a4bd6b54254a78c13a P 66d91f159ad2454f9979f9dc82b5a89b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:24:19.908772 9972 tablet_bootstrap.cc:492] T bc5f5036f46941a4bd6b54254a78c13a P 19abf35f684045d2ad0d70ce826b3863: No bootstrap required, opened a new log
I20260618 23:24:19.908816 9972 ts_tablet_manager.cc:1397] T bc5f5036f46941a4bd6b54254a78c13a P 19abf35f684045d2ad0d70ce826b3863: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:24:19.908952 9972 raft_consensus.cc:348] T bc5f5036f46941a4bd6b54254a78c13a P 19abf35f684045d2ad0d70ce826b3863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } }
I20260618 23:24:19.909030 9972 raft_consensus.cc:374] T bc5f5036f46941a4bd6b54254a78c13a P 19abf35f684045d2ad0d70ce826b3863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:19.909060 9972 raft_consensus.cc:729] T bc5f5036f46941a4bd6b54254a78c13a P 19abf35f684045d2ad0d70ce826b3863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19abf35f684045d2ad0d70ce826b3863, State: Initialized, Role: FOLLOWER
I20260618 23:24:19.909107 9972 consensus_queue.cc:260] T bc5f5036f46941a4bd6b54254a78c13a P 19abf35f684045d2ad0d70ce826b3863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } }
I20260618 23:24:19.909225 9972 ts_tablet_manager.cc:1428] T bc5f5036f46941a4bd6b54254a78c13a P 19abf35f684045d2ad0d70ce826b3863: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260618 23:24:19.909279 9975 tablet_replica.cc:439] T bc5f5036f46941a4bd6b54254a78c13a P 19abf35f684045d2ad0d70ce826b3863: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } } }
I20260618 23:24:19.909399 9975 tablet_replica.cc:442] T bc5f5036f46941a4bd6b54254a78c13a P 19abf35f684045d2ad0d70ce826b3863: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:24:19.932324 9977 raft_consensus.cc:482] T bc5f5036f46941a4bd6b54254a78c13a P 66d91f159ad2454f9979f9dc82b5a89b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:24:19.932406 9977 raft_consensus.cc:504] T bc5f5036f46941a4bd6b54254a78c13a P 66d91f159ad2454f9979f9dc82b5a89b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } }
I20260618 23:24:19.932890 9977 leader_election.cc:288] T bc5f5036f46941a4bd6b54254a78c13a P 66d91f159ad2454f9979f9dc82b5a89b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 19abf35f684045d2ad0d70ce826b3863 (127.8.39.129:38929), 79a1d312cc5644d893085e5bde1e9d64 (127.8.39.130:38953)
I20260618 23:24:19.935806 9967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.933219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:38889 (local address 127.8.39.129:38929)
0618 23:24:19.933299 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:19.933301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:19.933313 (+ 12us) server_negotiation.cc:407] Connection header received
0618 23:24:19.933344 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:19.933345 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:19.933354 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:19.933429 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:19.933572 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.934064 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.935077 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.935437 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.935492 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.935519 (+ 27us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:19.935693 (+ 174us) server_negotiation.cc:299] Negotiation successful
0618 23:24:19.935733 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260618 23:24:19.935812 9970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.933110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:50583 (local address 127.8.39.130:38953)
0618 23:24:19.933171 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:19.933173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:19.933275 (+ 102us) server_negotiation.cc:407] Connection header received
0618 23:24:19.933365 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:19.933365 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:19.933373 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:19.933427 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:19.933632 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.934068 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.935100 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.935444 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.935534 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.935542 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:19.935716 (+ 174us) server_negotiation.cc:299] Negotiation successful
0618 23:24:19.935750 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260618 23:24:19.935827 10012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.933102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:38929 (local address 127.8.39.131:38889)
0618 23:24:19.933229 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:19.933240 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:19.933291 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:19.933431 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:19.933432 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:19.933433 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:19.933492 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.933494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.934148 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.934149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.934981 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.934983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.935536 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.935537 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.935611 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.935617 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 23:24:19.935664 (+ 47us) client_negotiation.cc:240] Negotiation successful
0618 23:24:19.935724 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":33,"thread_start_us":89,"threads_started":1}
I20260618 23:24:19.935833 10011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.933070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:38953 (local address 127.8.39.131:50583)
0618 23:24:19.933209 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:19.933220 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:19.933294 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:19.933451 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:19.933452 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:19.933454 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:19.933525 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.933554 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.934173 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.934174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.934985 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.934987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.935557 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.935558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.935626 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.935629 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:24:19.935702 (+ 73us) client_negotiation.cc:240] Negotiation successful
0618 23:24:19.935739 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":48,"thread_start_us":90,"threads_started":1}
I20260618 23:24:19.936146 9622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc5f5036f46941a4bd6b54254a78c13a" candidate_uuid: "66d91f159ad2454f9979f9dc82b5a89b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19abf35f684045d2ad0d70ce826b3863" is_pre_election: true
I20260618 23:24:19.936290 9622 raft_consensus.cc:2454] T bc5f5036f46941a4bd6b54254a78c13a P 19abf35f684045d2ad0d70ce826b3863 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66d91f159ad2454f9979f9dc82b5a89b in term 0.
I20260618 23:24:19.936409 9762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc5f5036f46941a4bd6b54254a78c13a" candidate_uuid: "66d91f159ad2454f9979f9dc82b5a89b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79a1d312cc5644d893085e5bde1e9d64" is_pre_election: true
I20260618 23:24:19.936486 9762 raft_consensus.cc:2454] T bc5f5036f46941a4bd6b54254a78c13a P 79a1d312cc5644d893085e5bde1e9d64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 66d91f159ad2454f9979f9dc82b5a89b in term 0.
I20260618 23:24:19.936451 9835 leader_election.cc:302] T bc5f5036f46941a4bd6b54254a78c13a P 66d91f159ad2454f9979f9dc82b5a89b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 19abf35f684045d2ad0d70ce826b3863, 66d91f159ad2454f9979f9dc82b5a89b; no voters:
I20260618 23:24:19.936589 9977 raft_consensus.cc:2781] T bc5f5036f46941a4bd6b54254a78c13a P 66d91f159ad2454f9979f9dc82b5a89b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:24:19.936630 9977 raft_consensus.cc:482] T bc5f5036f46941a4bd6b54254a78c13a P 66d91f159ad2454f9979f9dc82b5a89b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:24:19.936685 9977 raft_consensus.cc:3037] T bc5f5036f46941a4bd6b54254a78c13a P 66d91f159ad2454f9979f9dc82b5a89b [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:19.937512 9977 raft_consensus.cc:504] T bc5f5036f46941a4bd6b54254a78c13a P 66d91f159ad2454f9979f9dc82b5a89b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } }
I20260618 23:24:19.937655 9977 leader_election.cc:288] T bc5f5036f46941a4bd6b54254a78c13a P 66d91f159ad2454f9979f9dc82b5a89b [CANDIDATE]: Term 1 election: Requested vote from peers 19abf35f684045d2ad0d70ce826b3863 (127.8.39.129:38929), 79a1d312cc5644d893085e5bde1e9d64 (127.8.39.130:38953)
I20260618 23:24:19.937800 9762 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc5f5036f46941a4bd6b54254a78c13a" candidate_uuid: "66d91f159ad2454f9979f9dc82b5a89b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79a1d312cc5644d893085e5bde1e9d64"
I20260618 23:24:19.937813 9622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bc5f5036f46941a4bd6b54254a78c13a" candidate_uuid: "66d91f159ad2454f9979f9dc82b5a89b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19abf35f684045d2ad0d70ce826b3863"
I20260618 23:24:19.937875 9622 raft_consensus.cc:3037] T bc5f5036f46941a4bd6b54254a78c13a P 19abf35f684045d2ad0d70ce826b3863 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:19.937875 9762 raft_consensus.cc:3037] T bc5f5036f46941a4bd6b54254a78c13a P 79a1d312cc5644d893085e5bde1e9d64 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:19.938588 9762 raft_consensus.cc:2454] T bc5f5036f46941a4bd6b54254a78c13a P 79a1d312cc5644d893085e5bde1e9d64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66d91f159ad2454f9979f9dc82b5a89b in term 1.
I20260618 23:24:19.938665 9622 raft_consensus.cc:2454] T bc5f5036f46941a4bd6b54254a78c13a P 19abf35f684045d2ad0d70ce826b3863 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 66d91f159ad2454f9979f9dc82b5a89b in term 1.
I20260618 23:24:19.938789 9833 leader_election.cc:302] T bc5f5036f46941a4bd6b54254a78c13a P 66d91f159ad2454f9979f9dc82b5a89b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 66d91f159ad2454f9979f9dc82b5a89b, 79a1d312cc5644d893085e5bde1e9d64; no voters:
I20260618 23:24:19.938905 9977 raft_consensus.cc:2781] T bc5f5036f46941a4bd6b54254a78c13a P 66d91f159ad2454f9979f9dc82b5a89b [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:19.939062 9977 raft_consensus.cc:686] T bc5f5036f46941a4bd6b54254a78c13a P 66d91f159ad2454f9979f9dc82b5a89b [term 1 LEADER]: Becoming Leader. State: Replica: 66d91f159ad2454f9979f9dc82b5a89b, State: Running, Role: LEADER
I20260618 23:24:19.939128 9977 consensus_queue.cc:237] T bc5f5036f46941a4bd6b54254a78c13a P 66d91f159ad2454f9979f9dc82b5a89b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } }
I20260618 23:24:19.939513 10013 tablet_replica.cc:439] T bc5f5036f46941a4bd6b54254a78c13a P 66d91f159ad2454f9979f9dc82b5a89b: TxnStatusTablet state changed. Reason: New leader 66d91f159ad2454f9979f9dc82b5a89b. Latest consensus state: current_term: 1 leader_uuid: "66d91f159ad2454f9979f9dc82b5a89b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } } }
I20260618 23:24:19.939618 10013 tablet_replica.cc:442] T bc5f5036f46941a4bd6b54254a78c13a P 66d91f159ad2454f9979f9dc82b5a89b: This TxnStatusTablet replica's current role is: LEADER
I20260618 23:24:19.939839 10015 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260618 23:24:19.939949 10015 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260618 23:24:19.939864 9465 catalog_manager.cc:5495] T bc5f5036f46941a4bd6b54254a78c13a P 66d91f159ad2454f9979f9dc82b5a89b reported cstate change: term changed from 0 to 1, leader changed from <none> to 66d91f159ad2454f9979f9dc82b5a89b (127.8.39.131). New cstate: current_term: 1 leader_uuid: "66d91f159ad2454f9979f9dc82b5a89b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } health_report { overall_health: HEALTHY } } }
I20260618 23:24:19.949448 9969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.946996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39478 (local address 127.8.39.131:44541)
0618 23:24:19.947041 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:19.947043 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:19.947051 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:24:19.947082 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:19.947083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:19.947099 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:19.947163 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:24:19.947309 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.947841 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.948588 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.948789 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.948884 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.949065 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:24:19.949219 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:24:19.949303 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:19.949359 (+ 56us) server_negotiation.cc:299] Negotiation successful
0618 23:24:19.949381 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260618 23:24:19.949471 10006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.946888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:44541 (local address 127.0.0.1:39478)
0618 23:24:19.946955 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:19.946967 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:19.947011 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:19.947168 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:19.947169 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:19.947170 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 23:24:19.947231 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.947233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.947917 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.947918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.948489 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.948491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.948873 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.948874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.948932 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.948947 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 23:24:19.949288 (+ 341us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 23:24:19.949297 (+ 9us) client_negotiation.cc:769] Sending connection context
0618 23:24:19.949354 (+ 57us) client_negotiation.cc:240] Negotiation successful
0618 23:24:19.949400 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260618 23:24:19.950448 9762 raft_consensus.cc:1261] T bc5f5036f46941a4bd6b54254a78c13a P 79a1d312cc5644d893085e5bde1e9d64 [term 1 FOLLOWER]: Refusing update from remote peer 66d91f159ad2454f9979f9dc82b5a89b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:24:19.950522 9622 raft_consensus.cc:1261] T bc5f5036f46941a4bd6b54254a78c13a P 19abf35f684045d2ad0d70ce826b3863 [term 1 FOLLOWER]: Refusing update from remote peer 66d91f159ad2454f9979f9dc82b5a89b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:24:19.950750 9977 consensus_queue.cc:1035] T bc5f5036f46941a4bd6b54254a78c13a P 66d91f159ad2454f9979f9dc82b5a89b [LEADER]: Connected to new peer: Peer: permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:19.950862 10013 consensus_queue.cc:1035] T bc5f5036f46941a4bd6b54254a78c13a P 66d91f159ad2454f9979f9dc82b5a89b [LEADER]: Connected to new peer: Peer: permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:19.951298 9979 tablet_replica.cc:439] T bc5f5036f46941a4bd6b54254a78c13a P 79a1d312cc5644d893085e5bde1e9d64: TxnStatusTablet state changed. Reason: New leader 66d91f159ad2454f9979f9dc82b5a89b. Latest consensus state: current_term: 1 leader_uuid: "66d91f159ad2454f9979f9dc82b5a89b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } } }
I20260618 23:24:19.951392 9979 tablet_replica.cc:442] T bc5f5036f46941a4bd6b54254a78c13a P 79a1d312cc5644d893085e5bde1e9d64: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:24:19.951584 9975 tablet_replica.cc:439] T bc5f5036f46941a4bd6b54254a78c13a P 19abf35f684045d2ad0d70ce826b3863: TxnStatusTablet state changed. Reason: New leader 66d91f159ad2454f9979f9dc82b5a89b. Latest consensus state: current_term: 1 leader_uuid: "66d91f159ad2454f9979f9dc82b5a89b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } } }
I20260618 23:24:19.951666 9975 tablet_replica.cc:442] T bc5f5036f46941a4bd6b54254a78c13a P 19abf35f684045d2ad0d70ce826b3863: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:24:19.952165 9977 tablet_replica.cc:439] T bc5f5036f46941a4bd6b54254a78c13a P 66d91f159ad2454f9979f9dc82b5a89b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "66d91f159ad2454f9979f9dc82b5a89b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } } }
I20260618 23:24:19.952260 9977 tablet_replica.cc:442] T bc5f5036f46941a4bd6b54254a78c13a P 66d91f159ad2454f9979f9dc82b5a89b: This TxnStatusTablet replica's current role is: LEADER
I20260618 23:24:19.952363 9975 tablet_replica.cc:439] T bc5f5036f46941a4bd6b54254a78c13a P 19abf35f684045d2ad0d70ce826b3863: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "66d91f159ad2454f9979f9dc82b5a89b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } } }
I20260618 23:24:19.952435 9975 tablet_replica.cc:442] T bc5f5036f46941a4bd6b54254a78c13a P 19abf35f684045d2ad0d70ce826b3863: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:24:19.953425 9977 tablet_replica.cc:439] T bc5f5036f46941a4bd6b54254a78c13a P 66d91f159ad2454f9979f9dc82b5a89b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "66d91f159ad2454f9979f9dc82b5a89b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } } }
I20260618 23:24:19.953523 9977 tablet_replica.cc:442] T bc5f5036f46941a4bd6b54254a78c13a P 66d91f159ad2454f9979f9dc82b5a89b: This TxnStatusTablet replica's current role is: LEADER
I20260618 23:24:19.953927 9979 tablet_replica.cc:439] T bc5f5036f46941a4bd6b54254a78c13a P 79a1d312cc5644d893085e5bde1e9d64: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "66d91f159ad2454f9979f9dc82b5a89b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "19abf35f684045d2ad0d70ce826b3863" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38929 } } peers { permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 } } peers { permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 } } }
I20260618 23:24:19.954008 9979 tablet_replica.cc:442] T bc5f5036f46941a4bd6b54254a78c13a P 79a1d312cc5644d893085e5bde1e9d64: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:24:19.958472 9969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.955773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:49565 (local address 127.8.39.131:44541)
0618 23:24:19.955814 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:19.955815 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:19.955917 (+ 102us) server_negotiation.cc:407] Connection header received
0618 23:24:19.955997 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:19.955997 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:19.956005 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:19.956099 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:24:19.956214 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.956642 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.957794 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.958044 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.958100 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.958274 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:24:19.958298 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:24:19.958337 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:19.958412 (+ 75us) server_negotiation.cc:299] Negotiation successful
0618 23:24:19.958426 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260618 23:24:19.958536 10023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.955690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:44541 (local address 127.8.39.129:49565)
0618 23:24:19.955857 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:19.955868 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:19.955933 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:19.956106 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:19.956107 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:19.956109 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 23:24:19.956156 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.956158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.956707 (+ 549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.956708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.957716 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.957719 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.958118 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.958119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.958177 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.958192 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 23:24:19.958356 (+ 164us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 23:24:19.958358 (+ 2us) client_negotiation.cc:769] Sending connection context
0618 23:24:19.958411 (+ 53us) client_negotiation.cc:240] Negotiation successful
0618 23:24:19.958457 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":85,"threads_started":1}
I20260618 23:24:19.960386 9902 raft_consensus.cc:1261] T f99e26e014184aa0845b1ac16bf638a7 P 66d91f159ad2454f9979f9dc82b5a89b [term 1 FOLLOWER]: Refusing update from remote peer 19abf35f684045d2ad0d70ce826b3863: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:24:19.960587 9762 raft_consensus.cc:1261] T f99e26e014184aa0845b1ac16bf638a7 P 79a1d312cc5644d893085e5bde1e9d64 [term 1 FOLLOWER]: Refusing update from remote peer 19abf35f684045d2ad0d70ce826b3863: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:24:19.960572 9982 consensus_queue.cc:1035] T f99e26e014184aa0845b1ac16bf638a7 P 19abf35f684045d2ad0d70ce826b3863 [LEADER]: Connected to new peer: Peer: permanent_uuid: "66d91f159ad2454f9979f9dc82b5a89b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:19.960827 9982 consensus_queue.cc:1035] T f99e26e014184aa0845b1ac16bf638a7 P 19abf35f684045d2ad0d70ce826b3863 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79a1d312cc5644d893085e5bde1e9d64" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38953 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:19.976210 9967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.967896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:40559 (local address 127.8.39.129:38929)
0618 23:24:19.967937 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:19.967939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:19.973267 (+ 5328us) server_negotiation.cc:407] Connection header received
0618 23:24:19.973305 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:19.973305 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:19.973317 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:19.973373 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:24:19.973590 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.974207 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.975417 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.975740 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.975802 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.975961 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:24:19.975995 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:24:19.976034 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:19.976114 (+ 80us) server_negotiation.cc:299] Negotiation successful
0618 23:24:19.976137 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260618 23:24:19.976260 10027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.967785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:38929 (local address 127.8.39.131:40559)
0618 23:24:19.973207 (+ 5422us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:19.973217 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:19.973271 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:19.973393 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:19.973394 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:19.973409 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 23:24:19.973467 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.973468 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.974288 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.974288 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.975340 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:19.975343 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.975799 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.975800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:19.975871 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.975885 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 23:24:19.976050 (+ 165us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 23:24:19.976051 (+ 1us) client_negotiation.cc:769] Sending connection context
0618 23:24:19.976096 (+ 45us) client_negotiation.cc:240] Negotiation successful
0618 23:24:19.976140 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5369,"thread_start_us":86,"threads_started":1}
I20260618 23:24:19.984027 9465 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:24:19.984135 9465 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:24:19.984700 9465 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ae35dc6bac64997a96505c30e9e6860: Sending DeleteTablet for 3 replicas of tablet f99e26e014184aa0845b1ac16bf638a7
I20260618 23:24:19.985208 9882 tablet_service.cc:1514] Processing DeleteTablet for tablet f99e26e014184aa0845b1ac16bf638a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:19 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:39466
I20260618 23:24:19.985244 9602 tablet_service.cc:1514] Processing DeleteTablet for tablet f99e26e014184aa0845b1ac16bf638a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:19 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:38152
I20260618 23:24:19.985074 9742 tablet_service.cc:1514] Processing DeleteTablet for tablet f99e26e014184aa0845b1ac16bf638a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:19 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:40658
I20260618 23:24:19.985296 9994 tablet_replica.cc:330] T f99e26e014184aa0845b1ac16bf638a7 P 66d91f159ad2454f9979f9dc82b5a89b: stopping tablet replica
I20260618 23:24:19.985337 9994 raft_consensus.cc:2229] T f99e26e014184aa0845b1ac16bf638a7 P 66d91f159ad2454f9979f9dc82b5a89b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:24:19.985342 9995 tablet_replica.cc:330] T f99e26e014184aa0845b1ac16bf638a7 P 19abf35f684045d2ad0d70ce826b3863: stopping tablet replica
I20260618 23:24:19.985349 9996 tablet_replica.cc:330] T f99e26e014184aa0845b1ac16bf638a7 P 79a1d312cc5644d893085e5bde1e9d64: stopping tablet replica
I20260618 23:24:19.985368 9994 raft_consensus.cc:2258] T f99e26e014184aa0845b1ac16bf638a7 P 66d91f159ad2454f9979f9dc82b5a89b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:19.985394 9996 raft_consensus.cc:2229] T f99e26e014184aa0845b1ac16bf638a7 P 79a1d312cc5644d893085e5bde1e9d64 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:24:19.985394 9995 raft_consensus.cc:2229] T f99e26e014184aa0845b1ac16bf638a7 P 19abf35f684045d2ad0d70ce826b3863 [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:24:19.985428 9996 raft_consensus.cc:2258] T f99e26e014184aa0845b1ac16bf638a7 P 79a1d312cc5644d893085e5bde1e9d64 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:19.985445 9995 raft_consensus.cc:2258] T f99e26e014184aa0845b1ac16bf638a7 P 19abf35f684045d2ad0d70ce826b3863 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:19.985683 9994 ts_tablet_manager.cc:1905] T f99e26e014184aa0845b1ac16bf638a7 P 66d91f159ad2454f9979f9dc82b5a89b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:19.985741 9995 ts_tablet_manager.cc:1905] T f99e26e014184aa0845b1ac16bf638a7 P 19abf35f684045d2ad0d70ce826b3863: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:19.986994 9995 ts_tablet_manager.cc:1918] T f99e26e014184aa0845b1ac16bf638a7 P 19abf35f684045d2ad0d70ce826b3863: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 23:24:19.987049 9995 log.cc:1196] T f99e26e014184aa0845b1ac16bf638a7 P 19abf35f684045d2ad0d70ce826b3863: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-0/wal/wals/f99e26e014184aa0845b1ac16bf638a7
I20260618 23:24:19.987218 9994 ts_tablet_manager.cc:1918] T f99e26e014184aa0845b1ac16bf638a7 P 66d91f159ad2454f9979f9dc82b5a89b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 23:24:19.987257 9995 ts_tablet_manager.cc:1939] T f99e26e014184aa0845b1ac16bf638a7 P 19abf35f684045d2ad0d70ce826b3863: Deleting consensus metadata
I20260618 23:24:19.987278 9994 log.cc:1196] T f99e26e014184aa0845b1ac16bf638a7 P 66d91f159ad2454f9979f9dc82b5a89b: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-2/wal/wals/f99e26e014184aa0845b1ac16bf638a7
I20260618 23:24:19.987484 9994 ts_tablet_manager.cc:1939] T f99e26e014184aa0845b1ac16bf638a7 P 66d91f159ad2454f9979f9dc82b5a89b: Deleting consensus metadata
I20260618 23:24:19.987519 9996 ts_tablet_manager.cc:1905] T f99e26e014184aa0845b1ac16bf638a7 P 79a1d312cc5644d893085e5bde1e9d64: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:19.987563 9451 catalog_manager.cc:4841] TS 19abf35f684045d2ad0d70ce826b3863 (127.8.39.129:38929): tablet f99e26e014184aa0845b1ac16bf638a7 (table test-table [id=792ca5cff30f405f9cef5cd5f61fec63]) successfully deleted
I20260618 23:24:19.989051 9996 ts_tablet_manager.cc:1918] T f99e26e014184aa0845b1ac16bf638a7 P 79a1d312cc5644d893085e5bde1e9d64: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 23:24:19.989157 9996 log.cc:1196] T f99e26e014184aa0845b1ac16bf638a7 P 79a1d312cc5644d893085e5bde1e9d64: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781825053542692-8350-0/minicluster-data/ts-1/wal/wals/f99e26e014184aa0845b1ac16bf638a7
I20260618 23:24:19.989399 9996 ts_tablet_manager.cc:1939] T f99e26e014184aa0845b1ac16bf638a7 P 79a1d312cc5644d893085e5bde1e9d64: Deleting consensus metadata
I20260618 23:24:19.989722 9449 catalog_manager.cc:4841] TS 79a1d312cc5644d893085e5bde1e9d64 (127.8.39.130:38953): tablet f99e26e014184aa0845b1ac16bf638a7 (table test-table [id=792ca5cff30f405f9cef5cd5f61fec63]) successfully deleted
I20260618 23:24:19.989986 9449 catalog_manager.cc:4841] TS 66d91f159ad2454f9979f9dc82b5a89b (127.8.39.131:44541): tablet f99e26e014184aa0845b1ac16bf638a7 (table test-table [id=792ca5cff30f405f9cef5cd5f61fec63]) successfully deleted
Jun 18 23:24:19 dist-test-slave-47qt krb5kdc[9420](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825059, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.39.129@KRBTEST.COM
I20260618 23:24:19.999015 9967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:19.988378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38164 (local address 127.8.39.129:38929)
0618 23:24:19.988415 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:19.988417 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:19.988551 (+ 134us) server_negotiation.cc:407] Connection header received
0618 23:24:19.988608 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:19.988608 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:19.988620 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:19.988685 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:19.989448 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.990190 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.991233 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:19.991540 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:19.991631 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:19.995208 (+ 3577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:19.995220 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:19.995220 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:19.995229 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:19.997595 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:19.997986 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:19.997989 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:19.997990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:19.998058 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:19.998243 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:19.998244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:19.998245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:19.998420 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:19.998580 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:19.998812 (+ 232us) server_negotiation.cc:299] Negotiation successful
0618 23:24:19.998915 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260618 23:24:19.999282 9662 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38164
I20260618 23:24:20.007166 10007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:20.000261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48666 (local address 127.8.39.190:34303)
0618 23:24:20.000307 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:20.000309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:20.000417 (+ 108us) server_negotiation.cc:407] Connection header received
0618 23:24:20.000450 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:20.000451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:20.000484 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:20.000543 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:20.001029 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:20.001882 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:20.002600 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:20.002929 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:20.003002 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:20.004237 (+ 1235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:20.004238 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:20.004239 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:20.004246 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:20.005899 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:20.006242 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:20.006243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:20.006243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:20.006300 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:20.006567 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:20.006567 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:20.006568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:20.006731 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:20.006837 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:20.006974 (+ 137us) server_negotiation.cc:299] Negotiation successful
0618 23:24:20.007057 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260618 23:24:20.007421 9465 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48666
I20260618 23:24:20.008073 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 9537
I20260618 23:24:20.017307 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 9677
I20260618 23:24:20.022270 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 9817
I20260618 23:24:20.027158 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 9433
2026-06-18T23:24:20Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2561 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:24:20 dist-test-slave-47qt krb5kdc[10036](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:24:20 dist-test-slave-47qt krb5kdc[10036](info): set up 2 sockets
Jun 18 23:24:20 dist-test-slave-47qt krb5kdc[10036](info): commencing operation
krb5kdc: starting...
W20260618 23:24:21.098158 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:24:21 dist-test-slave-47qt krb5kdc[10036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:24:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:24:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.39.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
I20260618 23:24:21.220294 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:33587
--webserver_interface=127.8.39.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:33587
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:21.300712 10049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:21.300978 10049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:21.301013 10049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:21.302163 10049 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:24:21.302241 10049 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260618 23:24:21.302264 10049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:21.302294 10049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:21.302323 10049 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:24:21.302341 10049 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:24:21.303823 10049 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:33587
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:33587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:21.304104 10049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:21.304488 10049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:21.310782 10054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:21.310802 10055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:21.310822 10057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:21.310894 10049 server_base.cc:1034] running on GCE node
I20260618 23:24:21.311277 10049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:21.311787 10049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:21.312965 10049 hybrid_clock.cc:648] HybridClock initialized: now 1781825061312941 us; error 43 us; skew 500 ppm
Jun 18 23:24:21 dist-test-slave-47qt krb5kdc[10036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825061, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:21.315805 10049 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:24:21.316473 10049 webserver.cc:460] Webserver started at http://127.8.39.190:42785/ using document root <none> and password file <none>
I20260618 23:24:21.316804 10049 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:21.316888 10049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:21.317250 10049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:21.318424 10049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "a7b49d74648e4730ba179de0645341e5"
format_stamp: "Formatted at 2026-06-18 23:24:21 on dist-test-slave-47qt"
server_key: "ce02f48526c21d75d557a6a5d5d8e561"
server_key_iv: "ca4f1664d9fa4c598c9dbdc2cba3e8f1"
server_key_version: "encryptionkey@0"
I20260618 23:24:21.318816 10049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "a7b49d74648e4730ba179de0645341e5"
format_stamp: "Formatted at 2026-06-18 23:24:21 on dist-test-slave-47qt"
server_key: "ce02f48526c21d75d557a6a5d5d8e561"
server_key_iv: "ca4f1664d9fa4c598c9dbdc2cba3e8f1"
server_key_version: "encryptionkey@0"
I20260618 23:24:21.320366 10049 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260618 23:24:21.321216 10064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:21.321547 10049 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:21.321785 10049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "a7b49d74648e4730ba179de0645341e5"
format_stamp: "Formatted at 2026-06-18 23:24:21 on dist-test-slave-47qt"
server_key: "ce02f48526c21d75d557a6a5d5d8e561"
server_key_iv: "ca4f1664d9fa4c598c9dbdc2cba3e8f1"
server_key_version: "encryptionkey@0"
I20260618 23:24:21.321884 10049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:21.329789 10049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:21.332381 10049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:21.332664 10049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:21.337366 10049 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:33587
I20260618 23:24:21.337421 10126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:33587 every 8 connection(s)
I20260618 23:24:21.337738 10049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:24:21.338362 10127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:21.340495 10127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7b49d74648e4730ba179de0645341e5: Bootstrap starting.
I20260618 23:24:21.341005 10127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7b49d74648e4730ba179de0645341e5: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:21.341336 10127 log.cc:824] T 00000000000000000000000000000000 P a7b49d74648e4730ba179de0645341e5: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:21.341988 10127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7b49d74648e4730ba179de0645341e5: No bootstrap required, opened a new log
I20260618 23:24:21.343966 10127 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7b49d74648e4730ba179de0645341e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b49d74648e4730ba179de0645341e5" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33587 } }
I20260618 23:24:21.344100 10127 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7b49d74648e4730ba179de0645341e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:21.344122 10127 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7b49d74648e4730ba179de0645341e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7b49d74648e4730ba179de0645341e5, State: Initialized, Role: FOLLOWER
I20260618 23:24:21.344262 10127 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7b49d74648e4730ba179de0645341e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b49d74648e4730ba179de0645341e5" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33587 } }
I20260618 23:24:21.344347 10127 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7b49d74648e4730ba179de0645341e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:24:21.344370 10127 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7b49d74648e4730ba179de0645341e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:24:21.344398 10127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7b49d74648e4730ba179de0645341e5 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:21.344877 10127 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7b49d74648e4730ba179de0645341e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b49d74648e4730ba179de0645341e5" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33587 } }
I20260618 23:24:21.344964 10127 leader_election.cc:302] T 00000000000000000000000000000000 P a7b49d74648e4730ba179de0645341e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7b49d74648e4730ba179de0645341e5; no voters:
I20260618 23:24:21.345031 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 10049
I20260618 23:24:21.345172 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:24:21.345193 10127 leader_election.cc:288] T 00000000000000000000000000000000 P a7b49d74648e4730ba179de0645341e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:24:21.345239 10130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7b49d74648e4730ba179de0645341e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:21.345348 8350 external_mini_cluster.cc:1428] Setting key e428deaf0ce8375fff7d8c8ffff2cf4b
I20260618 23:24:21.345369 10130 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7b49d74648e4730ba179de0645341e5 [term 1 LEADER]: Becoming Leader. State: Replica: a7b49d74648e4730ba179de0645341e5, State: Running, Role: LEADER
I20260618 23:24:21.345453 10127 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7b49d74648e4730ba179de0645341e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:24:21.345454 10130 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7b49d74648e4730ba179de0645341e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b49d74648e4730ba179de0645341e5" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33587 } }
I20260618 23:24:21.345901 10130 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7b49d74648e4730ba179de0645341e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7b49d74648e4730ba179de0645341e5. Latest consensus state: current_term: 1 leader_uuid: "a7b49d74648e4730ba179de0645341e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b49d74648e4730ba179de0645341e5" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33587 } } }
I20260618 23:24:21.346004 10130 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7b49d74648e4730ba179de0645341e5 [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:21.345911 10132 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7b49d74648e4730ba179de0645341e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7b49d74648e4730ba179de0645341e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b49d74648e4730ba179de0645341e5" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33587 } } }
I20260618 23:24:21.346199 10132 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7b49d74648e4730ba179de0645341e5 [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:21.346372 10140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:24:21.346962 10140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:24:21.348556 10140 catalog_manager.cc:1347] Generated new cluster ID: 4381a57a160647b888c970e633d94e45
I20260618 23:24:21.348613 10140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 18 23:24:21 dist-test-slave-47qt krb5kdc[10036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:21.357627 10138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:21.346009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37382 (local address 127.8.39.190:33587)
0618 23:24:21.346211 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:21.346235 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:21.346491 (+ 256us) server_negotiation.cc:407] Connection header received
0618 23:24:21.346908 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:21.346915 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:21.346973 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:21.347151 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:21.347957 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.349341 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.350404 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.350995 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.351087 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:21.353915 (+ 2828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:21.353921 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:21.353921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:21.353936 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:21.355950 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:21.356319 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:21.356322 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:21.356323 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:21.356441 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:21.356641 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:21.356642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:21.356642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:21.356975 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:21.357181 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:21.357253 (+ 72us) server_negotiation.cc:299] Negotiation successful
0618 23:24:21.357412 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":73,"threads_started":1}
I20260618 23:24:21.383235 10140 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:24:21.383828 10140 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:24:21.392828 10140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7b49d74648e4730ba179de0645341e5: Generated new TSK 0
I20260618 23:24:21.393055 10140 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.39.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
I20260618 23:24:21.420368 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.129:0
--local_ip_for_outbound_sockets=127.8.39.129
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:33587
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:21.507311 10153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:21.507503 10153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:21.507526 10153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:21.508661 10153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:21.508701 10153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:21.508721 10153 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260618 23:24:21.508733 10153 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 23:24:21.508750 10153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.129
I20260618 23:24:21.510232 10153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.39.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.39.190:33587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.129
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:21.510476 10153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:21.510730 10153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:21.515527 10159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:21.515554 10158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:21.515755 10153 server_base.cc:1034] running on GCE node
W20260618 23:24:21.515573 10161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:21.516033 10153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:21.516271 10153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:21.517413 10153 hybrid_clock.cc:648] HybridClock initialized: now 1781825061517401 us; error 26 us; skew 500 ppm
Jun 18 23:24:21 dist-test-slave-47qt krb5kdc[10036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825061, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:21.519780 10153 init.cc:377] Logged in from keytab as kudu/127.8.39.129@KRBTEST.COM (short username kudu)
I20260618 23:24:21.520152 10153 webserver.cc:460] Webserver started at http://127.8.39.129:33847/ using document root <none> and password file <none>
I20260618 23:24:21.520350 10153 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:21.520391 10153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:21.520475 10153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:21.521500 10153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-0/data/instance:
uuid: "1118c3f91fbe492cadf521766345707a"
format_stamp: "Formatted at 2026-06-18 23:24:21 on dist-test-slave-47qt"
server_key: "9f8a5a8ba143c31e0014b90d01b32b9e"
server_key_iv: "e84bcdf5d9b03e15c96dbbd5b2da134f"
server_key_version: "encryptionkey@0"
I20260618 23:24:21.521793 10153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance:
uuid: "1118c3f91fbe492cadf521766345707a"
format_stamp: "Formatted at 2026-06-18 23:24:21 on dist-test-slave-47qt"
server_key: "9f8a5a8ba143c31e0014b90d01b32b9e"
server_key_iv: "e84bcdf5d9b03e15c96dbbd5b2da134f"
server_key_version: "encryptionkey@0"
I20260618 23:24:21.523058 10153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:24:21.523818 10168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:21.524063 10153 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:21.524163 10153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-0/wal
uuid: "1118c3f91fbe492cadf521766345707a"
format_stamp: "Formatted at 2026-06-18 23:24:21 on dist-test-slave-47qt"
server_key: "9f8a5a8ba143c31e0014b90d01b32b9e"
server_key_iv: "e84bcdf5d9b03e15c96dbbd5b2da134f"
server_key_version: "encryptionkey@0"
I20260618 23:24:21.524250 10153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:21.534294 10153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:21.536561 10153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:21.536720 10153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:21.537256 10153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:21.537322 10153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:21.537359 10153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:21.537386 10153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:21.543908 10153 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.129:41901
I20260618 23:24:21.543937 10283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.129:41901 every 8 connection(s)
Jun 18 23:24:21 dist-test-slave-47qt krb5kdc[10036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825061, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:21.544312 10153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
I20260618 23:24:21.545239 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 10153
I20260618 23:24:21.545384 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance
I20260618 23:24:21.545573 8350 external_mini_cluster.cc:1428] Setting key b5a070a18b69e9342a3e93272b9901b4
I20260618 23:24:21.548153 10138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:21.537688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:40795 (local address 127.8.39.190:33587)
0618 23:24:21.537761 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:21.537764 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:21.537961 (+ 197us) server_negotiation.cc:407] Connection header received
0618 23:24:21.538422 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:21.538424 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:21.538440 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:21.538522 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:21.539305 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.540161 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.541702 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.542105 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.542188 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:21.544889 (+ 2701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:21.544890 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:21.544891 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:21.544897 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:21.546651 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:21.547073 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:21.547074 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:21.547074 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:21.547136 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:21.547483 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:21.547485 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:21.547485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:21.547610 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:21.547691 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:21.547981 (+ 290us) server_negotiation.cc:299] Negotiation successful
0618 23:24:21.548066 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 23:24:21.548326 10177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:21.537672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:33587 (local address 127.8.39.129:40795)
0618 23:24:21.537829 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:21.537866 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:21.538323 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:21.538569 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:21.538572 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:21.538919 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:21.539196 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:21.539203 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.540257 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.540258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:21.541609 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:21.541613 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.542193 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.542194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:21.542328 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:21.542861 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:21.542870 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:21.544798 (+ 1928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:21.546733 (+ 1935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:21.546739 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:21.546741 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:21.546994 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:21.547197 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:21.547198 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:21.547198 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:21.547401 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:21.547697 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:21.547703 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:21.547836 (+ 133us) client_negotiation.cc:769] Sending connection context
0618 23:24:21.547948 (+ 112us) client_negotiation.cc:240] Negotiation successful
0618 23:24:21.548103 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260618 23:24:21.548875 10284 heartbeater.cc:347] Connected to a master server at 127.8.39.190:33587
I20260618 23:24:21.548959 10284 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:21.549188 10284 heartbeater.cc:511] Master 127.8.39.190:33587 requested a full tablet report, sending...
I20260618 23:24:21.549690 10081 ts_manager.cc:194] Registered new tserver with Master: 1118c3f91fbe492cadf521766345707a (127.8.39.129:41901)
I20260618 23:24:21.550537 10081 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.129@KRBTEST.COM'} at 127.8.39.129:40795
WARNING: no policy specified for kudu/127.8.39.130@KRBTEST.COM; defaulting to no policy
I20260618 23:24:21.555794 10291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:21.550050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:33587 (local address 127.8.39.129:53641)
0618 23:24:21.550215 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:21.550228 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:21.550296 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:21.550425 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:21.550426 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:21.550625 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:21.550691 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:21.550693 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.551385 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.551386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:21.551896 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:21.551897 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.552264 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.552264 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:21.552344 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:21.552790 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:21.552795 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:21.553066 (+ 271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:21.554734 (+ 1668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:21.554735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:21.554735 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:21.554965 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:21.555170 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:21.555171 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:21.555171 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:21.555228 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:21.555475 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:21.555476 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:21.555520 (+ 44us) client_negotiation.cc:769] Sending connection context
0618 23:24:21.555605 (+ 85us) client_negotiation.cc:240] Negotiation successful
0618 23:24:21.555704 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":85,"threads_started":1}
I20260618 23:24:21.555835 10138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:21.549984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:53641 (local address 127.8.39.190:33587)
0618 23:24:21.550021 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:21.550022 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:21.550301 (+ 279us) server_negotiation.cc:407] Connection header received
0618 23:24:21.550357 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:21.550358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:21.550367 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:21.550419 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:21.550793 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.551284 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.551991 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.552191 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.552267 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:21.553220 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:21.553222 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:21.553222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:21.553229 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:21.554637 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:21.555047 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:21.555048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:21.555048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:21.555099 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:21.555297 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:21.555298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:21.555299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:21.555409 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:21.555483 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:21.555630 (+ 147us) server_negotiation.cc:299] Negotiation successful
0618 23:24:21.555702 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
I20260618 23:24:21.572844 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.130:0
--local_ip_for_outbound_sockets=127.8.39.130
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:33587
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:21.658144 10293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:21.658360 10293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:21.658385 10293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:21.659480 10293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:21.659519 10293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:21.659540 10293 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260618 23:24:21.659552 10293 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 23:24:21.659569 10293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.130
I20260618 23:24:21.661154 10293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.39.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.39.190:33587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.130
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:21.661372 10293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:21.661549 10293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:21.666635 10301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:21.666630 10298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:21.666630 10299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:21.666733 10293 server_base.cc:1034] running on GCE node
I20260618 23:24:21.667088 10293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:21.667331 10293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:21.668495 10293 hybrid_clock.cc:648] HybridClock initialized: now 1781825061668498 us; error 44 us; skew 500 ppm
Jun 18 23:24:21 dist-test-slave-47qt krb5kdc[10036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825061, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:21.671253 10293 init.cc:377] Logged in from keytab as kudu/127.8.39.130@KRBTEST.COM (short username kudu)
I20260618 23:24:21.671789 10293 webserver.cc:460] Webserver started at http://127.8.39.130:46131/ using document root <none> and password file <none>
I20260618 23:24:21.672022 10293 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:21.672091 10293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:21.672240 10293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:21.673349 10293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-1/data/instance:
uuid: "5a3086abd30a4331a257b50f39adf55e"
format_stamp: "Formatted at 2026-06-18 23:24:21 on dist-test-slave-47qt"
server_key: "8271d1398c42e1dc21dcedd65dd5671b"
server_key_iv: "88033ca8dfd2b3ea699940051c8b4aff"
server_key_version: "encryptionkey@0"
I20260618 23:24:21.673702 10293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance:
uuid: "5a3086abd30a4331a257b50f39adf55e"
format_stamp: "Formatted at 2026-06-18 23:24:21 on dist-test-slave-47qt"
server_key: "8271d1398c42e1dc21dcedd65dd5671b"
server_key_iv: "88033ca8dfd2b3ea699940051c8b4aff"
server_key_version: "encryptionkey@0"
I20260618 23:24:21.675014 10293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:24:21.675722 10308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:21.675989 10293 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:21.676110 10293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-1/wal
uuid: "5a3086abd30a4331a257b50f39adf55e"
format_stamp: "Formatted at 2026-06-18 23:24:21 on dist-test-slave-47qt"
server_key: "8271d1398c42e1dc21dcedd65dd5671b"
server_key_iv: "88033ca8dfd2b3ea699940051c8b4aff"
server_key_version: "encryptionkey@0"
I20260618 23:24:21.676195 10293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:21.701049 10293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:21.703588 10293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:21.703792 10293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:21.704452 10293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:21.704506 10293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:21.704541 10293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:21.704560 10293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 23:24:21 dist-test-slave-47qt krb5kdc[10036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825061, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:21.711752 10293 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.130:37285
I20260618 23:24:21.712165 10293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
I20260618 23:24:21.712193 10423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.130:37285 every 8 connection(s)
I20260618 23:24:21.715019 10138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:21.704927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:38413 (local address 127.8.39.190:33587)
0618 23:24:21.705003 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:21.705006 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:21.705316 (+ 310us) server_negotiation.cc:407] Connection header received
0618 23:24:21.706147 (+ 831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:21.706149 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:21.706161 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:21.706231 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:21.706851 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.707485 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.708541 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.708736 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.708829 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:21.711557 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:21.711558 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:21.711559 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:21.711567 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:21.713276 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:21.713926 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:21.713928 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:21.713928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:21.713994 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:21.714387 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:21.714388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:21.714388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:21.714508 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:21.714575 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:21.714889 (+ 314us) server_negotiation.cc:299] Negotiation successful
0618 23:24:21.714956 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260618 23:24:21.715118 10317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:21.704898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:33587 (local address 127.8.39.130:38413)
0618 23:24:21.705138 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:21.705189 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:21.705916 (+ 727us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:21.706308 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:21.706318 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:21.706549 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:21.706733 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:21.706737 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.707576 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.707578 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:21.708444 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:21.708446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.708841 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.708843 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:21.708979 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:21.709507 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:21.709518 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:21.711435 (+ 1917us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:21.713393 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:21.713406 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:21.713410 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:21.713794 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:21.714134 (+ 340us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:21.714135 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:21.714136 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:21.714292 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:21.714598 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:21.714609 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:21.714731 (+ 122us) client_negotiation.cc:769] Sending connection context
0618 23:24:21.714829 (+ 98us) client_negotiation.cc:240] Negotiation successful
0618 23:24:21.714967 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":114,"threads_started":1}
I20260618 23:24:21.715718 10424 heartbeater.cc:347] Connected to a master server at 127.8.39.190:33587
I20260618 23:24:21.715798 10424 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:21.715986 10424 heartbeater.cc:511] Master 127.8.39.190:33587 requested a full tablet report, sending...
I20260618 23:24:21.716439 10081 ts_manager.cc:194] Registered new tserver with Master: 5a3086abd30a4331a257b50f39adf55e (127.8.39.130:37285)
I20260618 23:24:21.716938 10081 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.130@KRBTEST.COM'} at 127.8.39.130:38413
I20260618 23:24:21.717936 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 10293
I20260618 23:24:21.718032 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance
I20260618 23:24:21.718204 8350 external_mini_cluster.cc:1428] Setting key a85bfb13a668cbf60bf6c7fc77ff4d31
I20260618 23:24:21.723170 10138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:21.716811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:43265 (local address 127.8.39.190:33587)
0618 23:24:21.716856 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:21.716858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:21.717043 (+ 185us) server_negotiation.cc:407] Connection header received
0618 23:24:21.717145 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:21.717145 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:21.717156 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:21.717205 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:21.717569 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.718255 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.719144 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.719479 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.719555 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:21.720507 (+ 952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:21.720508 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:21.720509 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:21.720515 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:21.721843 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:21.722253 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:21.722255 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:21.722255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:21.722313 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:21.722608 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:21.722608 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:21.722609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:21.722755 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:21.722816 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:21.723047 (+ 231us) server_negotiation.cc:299] Negotiation successful
0618 23:24:21.723113 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260618 23:24:21.723240 10430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:21.716738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:33587 (local address 127.8.39.130:43265)
0618 23:24:21.716937 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:21.716951 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:21.717073 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:21.717224 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:21.717225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:21.717423 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:21.717482 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:21.717484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.718356 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.718357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:21.719063 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:21.719065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.719576 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.719577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:21.719637 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:21.720126 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:21.720131 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:21.720424 (+ 293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:21.721981 (+ 1557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:21.721982 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:21.721982 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:21.722168 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:21.722423 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:21.722424 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:21.722424 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:21.722502 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:21.722833 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:21.722834 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:21.722885 (+ 51us) client_negotiation.cc:769] Sending connection context
0618 23:24:21.722966 (+ 81us) client_negotiation.cc:240] Negotiation successful
0618 23:24:21.723084 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for kudu/127.8.39.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
I20260618 23:24:21.748793 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.131:0
--local_ip_for_outbound_sockets=127.8.39.131
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:33587
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:21.831597 10433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:21.831849 10433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:21.831892 10433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:21.833056 10433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:21.833127 10433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:21.833195 10433 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260618 23:24:21.833217 10433 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 23:24:21.833256 10433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.131
I20260618 23:24:21.836000 10433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.39.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.39.190:33587
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10433
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.131
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:21.836318 10433 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:21.836565 10433 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:21.841730 10438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:21.841768 10439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:21.841769 10441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:21.842027 10433 server_base.cc:1034] running on GCE node
I20260618 23:24:21.842242 10433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:21.842501 10433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:21.843677 10433 hybrid_clock.cc:648] HybridClock initialized: now 1781825061843658 us; error 34 us; skew 500 ppm
Jun 18 23:24:21 dist-test-slave-47qt krb5kdc[10036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825061, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:21.847313 10433 init.cc:377] Logged in from keytab as kudu/127.8.39.131@KRBTEST.COM (short username kudu)
I20260618 23:24:21.848027 10433 webserver.cc:460] Webserver started at http://127.8.39.131:37169/ using document root <none> and password file <none>
I20260618 23:24:21.848327 10433 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:21.848407 10433 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:21.848559 10433 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:21.849954 10433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-2/data/instance:
uuid: "38ddfa85ddec470ca8a999e852c88dd9"
format_stamp: "Formatted at 2026-06-18 23:24:21 on dist-test-slave-47qt"
server_key: "ee9ccbab3dfcc6a751b331494288bdac"
server_key_iv: "b3c0bfaf83b25aa1c2f50709575e53c1"
server_key_version: "encryptionkey@0"
I20260618 23:24:21.850358 10433 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance:
uuid: "38ddfa85ddec470ca8a999e852c88dd9"
format_stamp: "Formatted at 2026-06-18 23:24:21 on dist-test-slave-47qt"
server_key: "ee9ccbab3dfcc6a751b331494288bdac"
server_key_iv: "b3c0bfaf83b25aa1c2f50709575e53c1"
server_key_version: "encryptionkey@0"
I20260618 23:24:21.851943 10433 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:24:21.852793 10448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:21.853224 10433 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:21.853487 10433 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-2/wal
uuid: "38ddfa85ddec470ca8a999e852c88dd9"
format_stamp: "Formatted at 2026-06-18 23:24:21 on dist-test-slave-47qt"
server_key: "ee9ccbab3dfcc6a751b331494288bdac"
server_key_iv: "b3c0bfaf83b25aa1c2f50709575e53c1"
server_key_version: "encryptionkey@0"
I20260618 23:24:21.853627 10433 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:21.873871 10433 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:21.876453 10433 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:21.876648 10433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:21.877274 10433 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:21.877326 10433 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:21.877360 10433 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:21.877378 10433 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 23:24:21 dist-test-slave-47qt krb5kdc[10036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825061, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:21.885197 10433 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.131:38833
I20260618 23:24:21.885428 10563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.131:38833 every 8 connection(s)
I20260618 23:24:21.885608 10433 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
I20260618 23:24:21.887912 10138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:21.877610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:59453 (local address 127.8.39.190:33587)
0618 23:24:21.877672 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:21.877674 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:21.877859 (+ 185us) server_negotiation.cc:407] Connection header received
0618 23:24:21.878409 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:21.878411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:21.878423 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:21.878486 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:21.879276 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.880101 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.881296 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.881499 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.881587 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:21.884830 (+ 3243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:21.884832 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:21.884833 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:21.884841 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:21.886445 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:21.886980 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:21.886981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:21.886981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:21.887037 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:21.887360 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:21.887362 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:21.887362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:21.887480 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:21.887546 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:21.887787 (+ 241us) server_negotiation.cc:299] Negotiation successful
0618 23:24:21.887850 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260618 23:24:21.888156 10457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:21.877602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:33587 (local address 127.8.39.131:59453)
0618 23:24:21.877745 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:21.877779 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:21.878299 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:21.878559 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:21.878563 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:21.878950 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:21.879188 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:21.879194 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.880197 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.880198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:21.881195 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:21.881198 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.881595 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.881596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:21.881728 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:21.882270 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:21.882282 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:21.884719 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:21.886538 (+ 1819us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:21.886543 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:21.886545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:21.886882 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:21.887098 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:21.887098 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:21.887099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:21.887189 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:21.887548 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:21.887553 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:21.887670 (+ 117us) client_negotiation.cc:769] Sending connection context
0618 23:24:21.887757 (+ 87us) client_negotiation.cc:240] Negotiation successful
0618 23:24:21.887890 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":70,"threads_started":1}
I20260618 23:24:21.888584 10564 heartbeater.cc:347] Connected to a master server at 127.8.39.190:33587
I20260618 23:24:21.888670 10564 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:21.888840 10564 heartbeater.cc:511] Master 127.8.39.190:33587 requested a full tablet report, sending...
I20260618 23:24:21.889243 10080 ts_manager.cc:194] Registered new tserver with Master: 38ddfa85ddec470ca8a999e852c88dd9 (127.8.39.131:38833)
I20260618 23:24:21.889719 10080 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.131@KRBTEST.COM'} at 127.8.39.131:59453
I20260618 23:24:21.894747 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 10433
I20260618 23:24:21.894833 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance
I20260618 23:24:21.895043 8350 external_mini_cluster.cc:1428] Setting key c4b6e18117d6ec8d7b991b6368a29786
I20260618 23:24:21.895057 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:21.889647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:33587 (local address 127.8.39.131:54083)
0618 23:24:21.889811 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:21.889822 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:21.889877 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:21.890008 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:21.890008 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:21.890151 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:21.890211 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:21.890212 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.890878 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.890878 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:21.891388 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:21.891389 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.891706 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.891707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:21.891757 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:21.892106 (+ 349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:21.892110 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:21.892362 (+ 252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:21.894068 (+ 1706us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:21.894069 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:21.894070 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:21.894263 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:21.894445 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:21.894445 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:21.894446 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:21.894496 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:21.894755 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:21.894755 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:21.894790 (+ 35us) client_negotiation.cc:769] Sending connection context
0618 23:24:21.894890 (+ 100us) client_negotiation.cc:240] Negotiation successful
0618 23:24:21.894988 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":75,"threads_started":1}
I20260618 23:24:21.895133 10138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:21.889726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:54083 (local address 127.8.39.190:33587)
0618 23:24:21.889764 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:21.889766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:21.889865 (+ 99us) server_negotiation.cc:407] Connection header received
0618 23:24:21.889925 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:21.889925 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:21.889935 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:21.889994 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:21.890286 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.890789 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.891461 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.891631 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.891685 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:21.892457 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:21.892458 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:21.892458 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:21.892466 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:21.893981 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:21.894335 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:21.894336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:21.894336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:21.894384 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:21.894534 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:21.894534 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:21.894535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:21.894689 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:21.894762 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:21.894969 (+ 207us) server_negotiation.cc:299] Negotiation successful
0618 23:24:21.895044 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260618 23:24:21.895829 8350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 18 23:24:21 dist-test-slave-47qt krb5kdc[10036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 18 23:24:21 dist-test-slave-47qt krb5kdc[10036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:21.919185 10138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:21.911353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37388 (local address 127.8.39.190:33587)
0618 23:24:21.911416 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:21.911418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:21.911426 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:24:21.911456 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:21.911457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:21.911468 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:21.911556 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:21.912120 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.912684 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.913734 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.914023 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.914103 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:21.916698 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:21.916700 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:21.916701 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:21.916712 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:21.918003 (+ 1291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:21.918401 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:21.918402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:21.918403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:21.918457 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:21.918675 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:21.918676 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:21.918676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:21.918785 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:21.918890 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:21.918996 (+ 106us) server_negotiation.cc:299] Negotiation successful
0618 23:24:21.919074 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260618 23:24:21.920831 10081 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:24:21.922410 10081 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:24:21.928913 10582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:21.924897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:41901 (local address 127.0.0.1:56268)
0618 23:24:21.925304 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:21.925313 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:21.925364 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:21.925644 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:21.925645 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:21.925647 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:21.925687 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:21.925688 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.926948 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.926949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:21.927897 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:21.927899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.928577 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.928578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:21.928658 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:21.928671 (+ 13us) client_negotiation.cc:769] Sending connection context
0618 23:24:21.928772 (+ 101us) client_negotiation.cc:240] Negotiation successful
0618 23:24:21.928827 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":62,"threads_started":1}
I20260618 23:24:21.928925 10586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:21.925054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56268 (local address 127.8.39.129:41901)
0618 23:24:21.925321 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:21.925325 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:21.925354 (+ 29us) server_negotiation.cc:407] Connection header received
0618 23:24:21.925487 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:21.925491 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:21.925508 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:21.925587 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:21.925763 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.926853 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.927975 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.928471 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.928572 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:21.928631 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:21.928789 (+ 158us) server_negotiation.cc:299] Negotiation successful
0618 23:24:21.928826 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":155,"threads_started":1}
I20260618 23:24:21.929061 10581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:21.924649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:37285 (local address 127.0.0.1:35636)
0618 23:24:21.924935 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:21.924976 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:21.925055 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:21.925333 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:21.925337 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:21.925339 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:21.925415 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:21.925418 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.926811 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.926812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:21.927926 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:21.927929 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.928847 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.928848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:21.928910 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:21.928914 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:24:21.928943 (+ 29us) client_negotiation.cc:240] Negotiation successful
0618 23:24:21.928988 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":93,"threads_started":1}
I20260618 23:24:21.929452 10585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:21.925021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48354 (local address 127.8.39.131:38833)
0618 23:24:21.925250 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:21.925254 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:21.925264 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:24:21.925359 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:21.925361 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:21.925376 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:21.925443 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:21.925626 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.926498 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.928060 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.928754 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.928818 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:21.928869 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:21.929098 (+ 229us) server_negotiation.cc:299] Negotiation successful
0618 23:24:21.929181 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":111,"threads_started":1}
I20260618 23:24:21.929728 10584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:21.924700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:38833 (local address 127.0.0.1:48354)
0618 23:24:21.925214 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:21.925224 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:21.925265 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:21.925496 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:21.925497 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:21.925499 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:21.925552 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:21.925554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.926577 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.926578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:21.927981 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:21.927983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.928895 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.928896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:21.928951 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:21.928953 (+ 2us) client_negotiation.cc:769] Sending connection context
0618 23:24:21.928996 (+ 43us) client_negotiation.cc:240] Negotiation successful
0618 23:24:21.929025 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"mutex_wait_us":43,"thread_start_us":85,"threads_started":1}
I20260618 23:24:21.929981 10583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:21.924762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35636 (local address 127.8.39.130:37285)
0618 23:24:21.924950 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:21.924955 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:21.925053 (+ 98us) server_negotiation.cc:407] Connection header received
0618 23:24:21.925193 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:21.925196 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:21.925214 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:21.925493 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:21.925544 (+ 51us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.926702 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.928038 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:21.928745 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:21.929519 (+ 774us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:21.929699 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:21.929729 (+ 30us) server_negotiation.cc:299] Negotiation successful
0618 23:24:21.929787 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":78,"threads_started":1}
I20260618 23:24:21.929870 10218 tablet_service.cc:1467] Processing CreateTablet for tablet c2d11ab17fda469f90ebdda33a27911a (DEFAULT_TABLE table=test-table [id=3d09a097908247ef914e50452b2443ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:21.930231 10218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2d11ab17fda469f90ebdda33a27911a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:21.930550 10358 tablet_service.cc:1467] Processing CreateTablet for tablet c2d11ab17fda469f90ebdda33a27911a (DEFAULT_TABLE table=test-table [id=3d09a097908247ef914e50452b2443ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:21.930824 10358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2d11ab17fda469f90ebdda33a27911a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:21.933274 10587 tablet_bootstrap.cc:492] T c2d11ab17fda469f90ebdda33a27911a P 1118c3f91fbe492cadf521766345707a: Bootstrap starting.
I20260618 23:24:21.933395 10498 tablet_service.cc:1467] Processing CreateTablet for tablet c2d11ab17fda469f90ebdda33a27911a (DEFAULT_TABLE table=test-table [id=3d09a097908247ef914e50452b2443ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:21.933548 10588 tablet_bootstrap.cc:492] T c2d11ab17fda469f90ebdda33a27911a P 5a3086abd30a4331a257b50f39adf55e: Bootstrap starting.
I20260618 23:24:21.933704 10498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2d11ab17fda469f90ebdda33a27911a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:21.933902 10587 tablet_bootstrap.cc:654] T c2d11ab17fda469f90ebdda33a27911a P 1118c3f91fbe492cadf521766345707a: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:21.934103 10588 tablet_bootstrap.cc:654] T c2d11ab17fda469f90ebdda33a27911a P 5a3086abd30a4331a257b50f39adf55e: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:21.934136 10587 log.cc:824] T c2d11ab17fda469f90ebdda33a27911a P 1118c3f91fbe492cadf521766345707a: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:21.934401 10588 log.cc:824] T c2d11ab17fda469f90ebdda33a27911a P 5a3086abd30a4331a257b50f39adf55e: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:21.934891 10587 tablet_bootstrap.cc:492] T c2d11ab17fda469f90ebdda33a27911a P 1118c3f91fbe492cadf521766345707a: No bootstrap required, opened a new log
I20260618 23:24:21.934974 10587 ts_tablet_manager.cc:1397] T c2d11ab17fda469f90ebdda33a27911a P 1118c3f91fbe492cadf521766345707a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:24:21.935134 10588 tablet_bootstrap.cc:492] T c2d11ab17fda469f90ebdda33a27911a P 5a3086abd30a4331a257b50f39adf55e: No bootstrap required, opened a new log
I20260618 23:24:21.935204 10588 ts_tablet_manager.cc:1397] T c2d11ab17fda469f90ebdda33a27911a P 5a3086abd30a4331a257b50f39adf55e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:24:21.936602 10587 raft_consensus.cc:348] T c2d11ab17fda469f90ebdda33a27911a P 1118c3f91fbe492cadf521766345707a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } } peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } }
I20260618 23:24:21.936761 10587 raft_consensus.cc:374] T c2d11ab17fda469f90ebdda33a27911a P 1118c3f91fbe492cadf521766345707a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:21.936803 10587 raft_consensus.cc:729] T c2d11ab17fda469f90ebdda33a27911a P 1118c3f91fbe492cadf521766345707a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1118c3f91fbe492cadf521766345707a, State: Initialized, Role: FOLLOWER
I20260618 23:24:21.937031 10587 consensus_queue.cc:260] T c2d11ab17fda469f90ebdda33a27911a P 1118c3f91fbe492cadf521766345707a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } } peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } }
I20260618 23:24:21.937270 10591 tablet_bootstrap.cc:492] T c2d11ab17fda469f90ebdda33a27911a P 38ddfa85ddec470ca8a999e852c88dd9: Bootstrap starting.
I20260618 23:24:21.937333 10587 ts_tablet_manager.cc:1428] T c2d11ab17fda469f90ebdda33a27911a P 1118c3f91fbe492cadf521766345707a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260618 23:24:21.937268 10588 raft_consensus.cc:348] T c2d11ab17fda469f90ebdda33a27911a P 5a3086abd30a4331a257b50f39adf55e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } } peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } }
I20260618 23:24:21.937448 10284 heartbeater.cc:503] Master 127.8.39.190:33587 was elected leader, sending a full tablet report...
I20260618 23:24:21.937453 10588 raft_consensus.cc:374] T c2d11ab17fda469f90ebdda33a27911a P 5a3086abd30a4331a257b50f39adf55e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:21.937493 10588 raft_consensus.cc:729] T c2d11ab17fda469f90ebdda33a27911a P 5a3086abd30a4331a257b50f39adf55e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a3086abd30a4331a257b50f39adf55e, State: Initialized, Role: FOLLOWER
I20260618 23:24:21.937635 10588 consensus_queue.cc:260] T c2d11ab17fda469f90ebdda33a27911a P 5a3086abd30a4331a257b50f39adf55e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } } peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } }
I20260618 23:24:21.937922 10588 ts_tablet_manager.cc:1428] T c2d11ab17fda469f90ebdda33a27911a P 5a3086abd30a4331a257b50f39adf55e: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260618 23:24:21.937943 10424 heartbeater.cc:503] Master 127.8.39.190:33587 was elected leader, sending a full tablet report...
I20260618 23:24:21.938149 10591 tablet_bootstrap.cc:654] T c2d11ab17fda469f90ebdda33a27911a P 38ddfa85ddec470ca8a999e852c88dd9: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:21.938470 10591 log.cc:824] T c2d11ab17fda469f90ebdda33a27911a P 38ddfa85ddec470ca8a999e852c88dd9: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:21.939138 10591 tablet_bootstrap.cc:492] T c2d11ab17fda469f90ebdda33a27911a P 38ddfa85ddec470ca8a999e852c88dd9: No bootstrap required, opened a new log
I20260618 23:24:21.939226 10591 ts_tablet_manager.cc:1397] T c2d11ab17fda469f90ebdda33a27911a P 38ddfa85ddec470ca8a999e852c88dd9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:24:21.940517 10591 raft_consensus.cc:348] T c2d11ab17fda469f90ebdda33a27911a P 38ddfa85ddec470ca8a999e852c88dd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } } peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } }
I20260618 23:24:21.940887 10591 raft_consensus.cc:374] T c2d11ab17fda469f90ebdda33a27911a P 38ddfa85ddec470ca8a999e852c88dd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:21.940984 10591 raft_consensus.cc:729] T c2d11ab17fda469f90ebdda33a27911a P 38ddfa85ddec470ca8a999e852c88dd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ddfa85ddec470ca8a999e852c88dd9, State: Initialized, Role: FOLLOWER
I20260618 23:24:21.941350 10591 consensus_queue.cc:260] T c2d11ab17fda469f90ebdda33a27911a P 38ddfa85ddec470ca8a999e852c88dd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } } peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } }
I20260618 23:24:21.941846 10564 heartbeater.cc:503] Master 127.8.39.190:33587 was elected leader, sending a full tablet report...
I20260618 23:24:21.941895 10591 ts_tablet_manager.cc:1428] T c2d11ab17fda469f90ebdda33a27911a P 38ddfa85ddec470ca8a999e852c88dd9: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260618 23:24:21.962954 10425 tablet.cc:2369] T c2d11ab17fda469f90ebdda33a27911a P 5a3086abd30a4331a257b50f39adf55e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:24:22.013574 10595 raft_consensus.cc:482] T c2d11ab17fda469f90ebdda33a27911a P 38ddfa85ddec470ca8a999e852c88dd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:24:22.013700 10595 raft_consensus.cc:504] T c2d11ab17fda469f90ebdda33a27911a P 38ddfa85ddec470ca8a999e852c88dd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } } peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } }
I20260618 23:24:22.014010 10595 leader_election.cc:288] T c2d11ab17fda469f90ebdda33a27911a P 38ddfa85ddec470ca8a999e852c88dd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a3086abd30a4331a257b50f39adf55e (127.8.39.130:37285), 1118c3f91fbe492cadf521766345707a (127.8.39.129:41901)
I20260618 23:24:22.017693 10586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:22.014339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:44053 (local address 127.8.39.129:41901)
0618 23:24:22.014383 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:22.014385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:22.014399 (+ 14us) server_negotiation.cc:407] Connection header received
0618 23:24:22.014549 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:22.014551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:22.014566 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:22.014653 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:22.014878 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.015399 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.016530 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.017178 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.017253 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:22.017262 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:22.017520 (+ 258us) server_negotiation.cc:299] Negotiation successful
0618 23:24:22.017593 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260618 23:24:22.017693 10457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:22.014291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:41901 (local address 127.8.39.131:44053)
0618 23:24:22.014351 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:22.014365 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:22.014416 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:22.014689 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:22.014690 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:22.014692 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:22.014788 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:22.014790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.015487 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.015488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:22.016403 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:22.016405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.017289 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.017291 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:22.017404 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:22.017411 (+ 7us) client_negotiation.cc:769] Sending connection context
0618 23:24:22.017505 (+ 94us) client_negotiation.cc:240] Negotiation successful
0618 23:24:22.017593 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260618 23:24:22.018080 10583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:22.014331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:50393 (local address 127.8.39.130:37285)
0618 23:24:22.014383 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:22.014385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:22.014673 (+ 288us) server_negotiation.cc:407] Connection header received
0618 23:24:22.014706 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:22.014707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:22.014720 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:22.014787 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:22.014996 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.015519 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.017098 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.017668 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.017712 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:22.017721 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:22.017965 (+ 244us) server_negotiation.cc:299] Negotiation successful
0618 23:24:22.018001 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260618 23:24:22.018100 10596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:22.014291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:37285 (local address 127.8.39.131:50393)
0618 23:24:22.014589 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:22.014601 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:22.014660 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:22.014795 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:22.014796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:22.014798 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:22.014900 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:22.014902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.015730 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.015732 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:22.016969 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:22.016971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.017803 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.017804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:22.017905 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:22.017912 (+ 7us) client_negotiation.cc:769] Sending connection context
0618 23:24:22.017960 (+ 48us) client_negotiation.cc:240] Negotiation successful
0618 23:24:22.018003 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":70,"thread_start_us":126,"threads_started":1}
I20260618 23:24:22.018198 10238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d11ab17fda469f90ebdda33a27911a" candidate_uuid: "38ddfa85ddec470ca8a999e852c88dd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1118c3f91fbe492cadf521766345707a" is_pre_election: true
I20260618 23:24:22.018420 10238 raft_consensus.cc:2454] T c2d11ab17fda469f90ebdda33a27911a P 1118c3f91fbe492cadf521766345707a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38ddfa85ddec470ca8a999e852c88dd9 in term 0.
I20260618 23:24:22.018600 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d11ab17fda469f90ebdda33a27911a" candidate_uuid: "38ddfa85ddec470ca8a999e852c88dd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a3086abd30a4331a257b50f39adf55e" is_pre_election: true
I20260618 23:24:22.018793 10378 raft_consensus.cc:2454] T c2d11ab17fda469f90ebdda33a27911a P 5a3086abd30a4331a257b50f39adf55e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38ddfa85ddec470ca8a999e852c88dd9 in term 0.
I20260618 23:24:22.018766 10451 leader_election.cc:302] T c2d11ab17fda469f90ebdda33a27911a P 38ddfa85ddec470ca8a999e852c88dd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1118c3f91fbe492cadf521766345707a, 38ddfa85ddec470ca8a999e852c88dd9; no voters:
I20260618 23:24:22.018956 10595 raft_consensus.cc:2781] T c2d11ab17fda469f90ebdda33a27911a P 38ddfa85ddec470ca8a999e852c88dd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:24:22.019042 10595 raft_consensus.cc:482] T c2d11ab17fda469f90ebdda33a27911a P 38ddfa85ddec470ca8a999e852c88dd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:24:22.019071 10595 raft_consensus.cc:3037] T c2d11ab17fda469f90ebdda33a27911a P 38ddfa85ddec470ca8a999e852c88dd9 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:22.019945 10595 raft_consensus.cc:504] T c2d11ab17fda469f90ebdda33a27911a P 38ddfa85ddec470ca8a999e852c88dd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } } peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } }
I20260618 23:24:22.020308 10595 leader_election.cc:288] T c2d11ab17fda469f90ebdda33a27911a P 38ddfa85ddec470ca8a999e852c88dd9 [CANDIDATE]: Term 1 election: Requested vote from peers 5a3086abd30a4331a257b50f39adf55e (127.8.39.130:37285), 1118c3f91fbe492cadf521766345707a (127.8.39.129:41901)
I20260618 23:24:22.020535 10238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d11ab17fda469f90ebdda33a27911a" candidate_uuid: "38ddfa85ddec470ca8a999e852c88dd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1118c3f91fbe492cadf521766345707a"
I20260618 23:24:22.020604 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2d11ab17fda469f90ebdda33a27911a" candidate_uuid: "38ddfa85ddec470ca8a999e852c88dd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a3086abd30a4331a257b50f39adf55e"
I20260618 23:24:22.020654 10238 raft_consensus.cc:3037] T c2d11ab17fda469f90ebdda33a27911a P 1118c3f91fbe492cadf521766345707a [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:22.020709 10378 raft_consensus.cc:3037] T c2d11ab17fda469f90ebdda33a27911a P 5a3086abd30a4331a257b50f39adf55e [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:22.021533 10238 raft_consensus.cc:2454] T c2d11ab17fda469f90ebdda33a27911a P 1118c3f91fbe492cadf521766345707a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38ddfa85ddec470ca8a999e852c88dd9 in term 1.
I20260618 23:24:22.021539 10378 raft_consensus.cc:2454] T c2d11ab17fda469f90ebdda33a27911a P 5a3086abd30a4331a257b50f39adf55e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38ddfa85ddec470ca8a999e852c88dd9 in term 1.
I20260618 23:24:22.021713 10451 leader_election.cc:302] T c2d11ab17fda469f90ebdda33a27911a P 38ddfa85ddec470ca8a999e852c88dd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1118c3f91fbe492cadf521766345707a, 38ddfa85ddec470ca8a999e852c88dd9; no voters:
I20260618 23:24:22.021847 10595 raft_consensus.cc:2781] T c2d11ab17fda469f90ebdda33a27911a P 38ddfa85ddec470ca8a999e852c88dd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:22.022039 10595 raft_consensus.cc:686] T c2d11ab17fda469f90ebdda33a27911a P 38ddfa85ddec470ca8a999e852c88dd9 [term 1 LEADER]: Becoming Leader. State: Replica: 38ddfa85ddec470ca8a999e852c88dd9, State: Running, Role: LEADER
I20260618 23:24:22.022147 10595 consensus_queue.cc:237] T c2d11ab17fda469f90ebdda33a27911a P 38ddfa85ddec470ca8a999e852c88dd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } } peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } }
I20260618 23:24:22.022974 10080 catalog_manager.cc:5495] T c2d11ab17fda469f90ebdda33a27911a P 38ddfa85ddec470ca8a999e852c88dd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38ddfa85ddec470ca8a999e852c88dd9 (127.8.39.131). New cstate: current_term: 1 leader_uuid: "38ddfa85ddec470ca8a999e852c88dd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } health_report { overall_health: UNKNOWN } } }
W20260618 23:24:22.045023 10285 tablet.cc:2369] T c2d11ab17fda469f90ebdda33a27911a P 1118c3f91fbe492cadf521766345707a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 18 23:24:22 dist-test-slave-47qt krb5kdc[10036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825061, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:22.073863 10138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:22.065995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37390 (local address 127.8.39.190:33587)
0618 23:24:22.066047 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:22.066049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:22.066138 (+ 89us) server_negotiation.cc:407] Connection header received
0618 23:24:22.066205 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:22.066206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:22.066216 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:22.066273 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:22.066710 (+ 437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.067203 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.067897 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.068093 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.068176 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:22.070774 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:22.070775 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:22.070776 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:22.070783 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:22.072503 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:22.072986 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:22.072987 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:22.072987 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:22.073050 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:22.073290 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:22.073292 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:22.073292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:22.073411 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:22.073497 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:22.073657 (+ 160us) server_negotiation.cc:299] Negotiation successful
0618 23:24:22.073749 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260618 23:24:22.074014 10604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:22.065932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:33587 (local address 127.0.0.1:37390)
0618 23:24:22.066055 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:22.066065 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:22.066142 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:22.066278 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:22.066279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:22.066578 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:22.066622 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:22.066624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.067286 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.067288 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:22.067793 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:22.067794 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.068177 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.068178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:22.068229 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:22.068809 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:22.068813 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:22.070657 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:22.072650 (+ 1993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:22.072657 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:22.072660 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:22.072883 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:22.073153 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:22.073154 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:22.073154 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:22.073217 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:22.073486 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:22.073493 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:22.073544 (+ 51us) client_negotiation.cc:769] Sending connection context
0618 23:24:22.073635 (+ 91us) client_negotiation.cc:240] Negotiation successful
0618 23:24:22.073848 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"mutex_wait_us":104,"thread_start_us":61,"threads_started":1}
I20260618 23:24:22.076071 10080 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:37390:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260618 23:24:22.076292 10080 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:24:22.078095 10218 tablet_service.cc:1467] Processing CreateTablet for tablet 9cfe5089d0a546c99f45094690948d4e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=63fc083309c64a1fb0a505974a2f1a24]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 23:24:22.078094 10358 tablet_service.cc:1467] Processing CreateTablet for tablet 9cfe5089d0a546c99f45094690948d4e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=63fc083309c64a1fb0a505974a2f1a24]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 23:24:22.078270 10218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cfe5089d0a546c99f45094690948d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:22.078279 10358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cfe5089d0a546c99f45094690948d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:22.078469 10498 tablet_service.cc:1467] Processing CreateTablet for tablet 9cfe5089d0a546c99f45094690948d4e (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=63fc083309c64a1fb0a505974a2f1a24]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 23:24:22.078639 10498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9cfe5089d0a546c99f45094690948d4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:22.079792 10587 tablet_bootstrap.cc:492] T 9cfe5089d0a546c99f45094690948d4e P 1118c3f91fbe492cadf521766345707a: Bootstrap starting.
I20260618 23:24:22.079792 10588 tablet_bootstrap.cc:492] T 9cfe5089d0a546c99f45094690948d4e P 5a3086abd30a4331a257b50f39adf55e: Bootstrap starting.
I20260618 23:24:22.079986 10591 tablet_bootstrap.cc:492] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9: Bootstrap starting.
I20260618 23:24:22.080207 10587 tablet_bootstrap.cc:654] T 9cfe5089d0a546c99f45094690948d4e P 1118c3f91fbe492cadf521766345707a: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:22.080451 10591 tablet_bootstrap.cc:654] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:22.080464 10588 tablet_bootstrap.cc:654] T 9cfe5089d0a546c99f45094690948d4e P 5a3086abd30a4331a257b50f39adf55e: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:22.080874 10587 tablet_bootstrap.cc:492] T 9cfe5089d0a546c99f45094690948d4e P 1118c3f91fbe492cadf521766345707a: No bootstrap required, opened a new log
I20260618 23:24:22.080929 10587 ts_tablet_manager.cc:1397] T 9cfe5089d0a546c99f45094690948d4e P 1118c3f91fbe492cadf521766345707a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:22.081089 10587 raft_consensus.cc:348] T 9cfe5089d0a546c99f45094690948d4e P 1118c3f91fbe492cadf521766345707a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:22.081166 10591 tablet_bootstrap.cc:492] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9: No bootstrap required, opened a new log
I20260618 23:24:22.081239 10587 raft_consensus.cc:374] T 9cfe5089d0a546c99f45094690948d4e P 1118c3f91fbe492cadf521766345707a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:22.081270 10587 raft_consensus.cc:729] T 9cfe5089d0a546c99f45094690948d4e P 1118c3f91fbe492cadf521766345707a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1118c3f91fbe492cadf521766345707a, State: Initialized, Role: FOLLOWER
I20260618 23:24:22.081272 10591 ts_tablet_manager.cc:1397] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:22.081332 10588 tablet_bootstrap.cc:492] T 9cfe5089d0a546c99f45094690948d4e P 5a3086abd30a4331a257b50f39adf55e: No bootstrap required, opened a new log
I20260618 23:24:22.081323 10587 consensus_queue.cc:260] T 9cfe5089d0a546c99f45094690948d4e P 1118c3f91fbe492cadf521766345707a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:22.081378 10588 ts_tablet_manager.cc:1397] T 9cfe5089d0a546c99f45094690948d4e P 5a3086abd30a4331a257b50f39adf55e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260618 23:24:22.081452 10587 ts_tablet_manager.cc:1428] T 9cfe5089d0a546c99f45094690948d4e P 1118c3f91fbe492cadf521766345707a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:22.081409 10591 raft_consensus.cc:348] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:22.081487 10591 raft_consensus.cc:374] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:22.081512 10591 raft_consensus.cc:729] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ddfa85ddec470ca8a999e852c88dd9, State: Initialized, Role: FOLLOWER
I20260618 23:24:22.081488 10592 tablet_replica.cc:439] T 9cfe5089d0a546c99f45094690948d4e P 1118c3f91fbe492cadf521766345707a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } } }
I20260618 23:24:22.081517 10588 raft_consensus.cc:348] T 9cfe5089d0a546c99f45094690948d4e P 5a3086abd30a4331a257b50f39adf55e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:22.081552 10591 consensus_queue.cc:260] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:22.081574 10588 raft_consensus.cc:374] T 9cfe5089d0a546c99f45094690948d4e P 5a3086abd30a4331a257b50f39adf55e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:22.081574 10592 tablet_replica.cc:442] T 9cfe5089d0a546c99f45094690948d4e P 1118c3f91fbe492cadf521766345707a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:24:22.081601 10588 raft_consensus.cc:729] T 9cfe5089d0a546c99f45094690948d4e P 5a3086abd30a4331a257b50f39adf55e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a3086abd30a4331a257b50f39adf55e, State: Initialized, Role: FOLLOWER
I20260618 23:24:22.081645 10591 ts_tablet_manager.cc:1428] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:22.081663 10588 consensus_queue.cc:260] T 9cfe5089d0a546c99f45094690948d4e P 5a3086abd30a4331a257b50f39adf55e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:22.081702 10595 tablet_replica.cc:439] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } } }
I20260618 23:24:22.081763 10588 ts_tablet_manager.cc:1428] T 9cfe5089d0a546c99f45094690948d4e P 5a3086abd30a4331a257b50f39adf55e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:22.081773 10595 tablet_replica.cc:442] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:24:22.081911 10593 tablet_replica.cc:439] T 9cfe5089d0a546c99f45094690948d4e P 5a3086abd30a4331a257b50f39adf55e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } } }
I20260618 23:24:22.082000 10593 tablet_replica.cc:442] T 9cfe5089d0a546c99f45094690948d4e P 5a3086abd30a4331a257b50f39adf55e: This TxnStatusTablet replica's current role is: FOLLOWER
W20260618 23:24:22.136263 10565 tablet.cc:2369] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:24:22.293179 10595 raft_consensus.cc:482] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:24:22.293278 10595 raft_consensus.cc:504] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:22.293439 10595 leader_election.cc:288] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1118c3f91fbe492cadf521766345707a (127.8.39.129:41901), 5a3086abd30a4331a257b50f39adf55e (127.8.39.130:37285)
I20260618 23:24:22.293685 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cfe5089d0a546c99f45094690948d4e" candidate_uuid: "38ddfa85ddec470ca8a999e852c88dd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a3086abd30a4331a257b50f39adf55e" is_pre_election: true
I20260618 23:24:22.293699 10238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cfe5089d0a546c99f45094690948d4e" candidate_uuid: "38ddfa85ddec470ca8a999e852c88dd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1118c3f91fbe492cadf521766345707a" is_pre_election: true
I20260618 23:24:22.293771 10238 raft_consensus.cc:2454] T 9cfe5089d0a546c99f45094690948d4e P 1118c3f91fbe492cadf521766345707a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38ddfa85ddec470ca8a999e852c88dd9 in term 0.
I20260618 23:24:22.293793 10378 raft_consensus.cc:2454] T 9cfe5089d0a546c99f45094690948d4e P 5a3086abd30a4331a257b50f39adf55e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38ddfa85ddec470ca8a999e852c88dd9 in term 0.
I20260618 23:24:22.293910 10451 leader_election.cc:302] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1118c3f91fbe492cadf521766345707a, 38ddfa85ddec470ca8a999e852c88dd9; no voters:
I20260618 23:24:22.294004 10595 raft_consensus.cc:2781] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:24:22.294044 10595 raft_consensus.cc:482] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:24:22.294070 10595 raft_consensus.cc:3037] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:22.294708 10595 raft_consensus.cc:504] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:22.294837 10595 leader_election.cc:288] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9 [CANDIDATE]: Term 1 election: Requested vote from peers 1118c3f91fbe492cadf521766345707a (127.8.39.129:41901), 5a3086abd30a4331a257b50f39adf55e (127.8.39.130:37285)
I20260618 23:24:22.294994 10238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cfe5089d0a546c99f45094690948d4e" candidate_uuid: "38ddfa85ddec470ca8a999e852c88dd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1118c3f91fbe492cadf521766345707a"
I20260618 23:24:22.295068 10238 raft_consensus.cc:3037] T 9cfe5089d0a546c99f45094690948d4e P 1118c3f91fbe492cadf521766345707a [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:22.295053 10378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9cfe5089d0a546c99f45094690948d4e" candidate_uuid: "38ddfa85ddec470ca8a999e852c88dd9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a3086abd30a4331a257b50f39adf55e"
I20260618 23:24:22.295120 10378 raft_consensus.cc:3037] T 9cfe5089d0a546c99f45094690948d4e P 5a3086abd30a4331a257b50f39adf55e [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:22.295725 10378 raft_consensus.cc:2454] T 9cfe5089d0a546c99f45094690948d4e P 5a3086abd30a4331a257b50f39adf55e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38ddfa85ddec470ca8a999e852c88dd9 in term 1.
I20260618 23:24:22.295754 10238 raft_consensus.cc:2454] T 9cfe5089d0a546c99f45094690948d4e P 1118c3f91fbe492cadf521766345707a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38ddfa85ddec470ca8a999e852c88dd9 in term 1.
I20260618 23:24:22.295919 10450 leader_election.cc:302] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38ddfa85ddec470ca8a999e852c88dd9, 5a3086abd30a4331a257b50f39adf55e; no voters:
I20260618 23:24:22.296010 10595 raft_consensus.cc:2781] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:22.296092 10595 raft_consensus.cc:686] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9 [term 1 LEADER]: Becoming Leader. State: Replica: 38ddfa85ddec470ca8a999e852c88dd9, State: Running, Role: LEADER
I20260618 23:24:22.296152 10595 consensus_queue.cc:237] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:22.296607 10597 tablet_replica.cc:439] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9: TxnStatusTablet state changed. Reason: New leader 38ddfa85ddec470ca8a999e852c88dd9. Latest consensus state: current_term: 1 leader_uuid: "38ddfa85ddec470ca8a999e852c88dd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } } }
I20260618 23:24:22.296778 10597 tablet_replica.cc:442] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9: This TxnStatusTablet replica's current role is: LEADER
I20260618 23:24:22.297094 10609 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260618 23:24:22.297058 10079 catalog_manager.cc:5495] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 38ddfa85ddec470ca8a999e852c88dd9 (127.8.39.131). New cstate: current_term: 1 leader_uuid: "38ddfa85ddec470ca8a999e852c88dd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } health_report { overall_health: HEALTHY } } }
I20260618 23:24:22.297247 10609 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260618 23:24:22.315966 10604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:22.313054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:38833 (local address 127.0.0.1:48366)
0618 23:24:22.313173 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:22.313186 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:22.313239 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:22.313410 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:22.313411 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:22.313412 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 23:24:22.313454 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:22.313455 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.314346 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.314347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:22.314889 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:22.314890 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.315372 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.315373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:22.315421 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:22.315434 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 23:24:22.315763 (+ 329us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 23:24:22.315773 (+ 10us) client_negotiation.cc:769] Sending connection context
0618 23:24:22.315828 (+ 55us) client_negotiation.cc:240] Negotiation successful
0618 23:24:22.315880 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260618 23:24:22.316159 10585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:22.313165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48366 (local address 127.8.39.131:38833)
0618 23:24:22.313210 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:22.313212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:22.313236 (+ 24us) server_negotiation.cc:407] Connection header received
0618 23:24:22.313316 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:22.313316 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:22.313336 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:22.313401 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:24:22.313524 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.314264 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.314972 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.315274 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.315356 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:22.315524 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:24:22.315659 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:24:22.315734 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:22.316069 (+ 335us) server_negotiation.cc:299] Negotiation successful
0618 23:24:22.316097 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260618 23:24:22.317777 10378 raft_consensus.cc:1261] T 9cfe5089d0a546c99f45094690948d4e P 5a3086abd30a4331a257b50f39adf55e [term 1 FOLLOWER]: Refusing update from remote peer 38ddfa85ddec470ca8a999e852c88dd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:24:22.317817 10238 raft_consensus.cc:1261] T 9cfe5089d0a546c99f45094690948d4e P 1118c3f91fbe492cadf521766345707a [term 1 FOLLOWER]: Refusing update from remote peer 38ddfa85ddec470ca8a999e852c88dd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:24:22.318148 10597 consensus_queue.cc:1035] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:22.319315 10592 tablet_replica.cc:439] T 9cfe5089d0a546c99f45094690948d4e P 1118c3f91fbe492cadf521766345707a: TxnStatusTablet state changed. Reason: New leader 38ddfa85ddec470ca8a999e852c88dd9. Latest consensus state: current_term: 1 leader_uuid: "38ddfa85ddec470ca8a999e852c88dd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } } }
I20260618 23:24:22.319407 10592 tablet_replica.cc:442] T 9cfe5089d0a546c99f45094690948d4e P 1118c3f91fbe492cadf521766345707a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:24:22.319460 10597 consensus_queue.cc:1035] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:22.320163 10597 tablet_replica.cc:439] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "38ddfa85ddec470ca8a999e852c88dd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } } }
I20260618 23:24:22.320236 10597 tablet_replica.cc:442] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9: This TxnStatusTablet replica's current role is: LEADER
I20260618 23:24:22.320299 10612 mvcc.cc:205] Tried to move back new op lower bound from 7298355455251357696 to 7298355455165657088. Current Snapshot: MvccSnapshot[applied={T|T < 7298355455251357696}]
I20260618 23:24:22.320626 10593 tablet_replica.cc:439] T 9cfe5089d0a546c99f45094690948d4e P 5a3086abd30a4331a257b50f39adf55e: TxnStatusTablet state changed. Reason: New leader 38ddfa85ddec470ca8a999e852c88dd9. Latest consensus state: current_term: 1 leader_uuid: "38ddfa85ddec470ca8a999e852c88dd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } } }
I20260618 23:24:22.320713 10593 tablet_replica.cc:442] T 9cfe5089d0a546c99f45094690948d4e P 5a3086abd30a4331a257b50f39adf55e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:24:22.320941 10592 tablet_replica.cc:439] T 9cfe5089d0a546c99f45094690948d4e P 1118c3f91fbe492cadf521766345707a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "38ddfa85ddec470ca8a999e852c88dd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } } }
I20260618 23:24:22.321010 10592 tablet_replica.cc:442] T 9cfe5089d0a546c99f45094690948d4e P 1118c3f91fbe492cadf521766345707a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:24:22.322324 10595 tablet_replica.cc:439] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "38ddfa85ddec470ca8a999e852c88dd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } } }
I20260618 23:24:22.322408 10595 tablet_replica.cc:442] T 9cfe5089d0a546c99f45094690948d4e P 38ddfa85ddec470ca8a999e852c88dd9: This TxnStatusTablet replica's current role is: LEADER
I20260618 23:24:22.322849 10593 tablet_replica.cc:439] T 9cfe5089d0a546c99f45094690948d4e P 5a3086abd30a4331a257b50f39adf55e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "38ddfa85ddec470ca8a999e852c88dd9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } } }
I20260618 23:24:22.322939 10593 tablet_replica.cc:442] T 9cfe5089d0a546c99f45094690948d4e P 5a3086abd30a4331a257b50f39adf55e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:24:22.325579 10585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:22.323307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48374 (local address 127.8.39.131:38833)
0618 23:24:22.323346 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:22.323348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:22.323357 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:24:22.323417 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:22.323418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:22.323427 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:22.323480 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:24:22.323607 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.324055 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.324894 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.325100 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.325204 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:22.325359 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:24:22.325389 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:24:22.325436 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:22.325490 (+ 54us) server_negotiation.cc:299] Negotiation successful
0618 23:24:22.325508 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260618 23:24:22.330960 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:22.328219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:38833 (local address 127.8.39.131:45597)
0618 23:24:22.328290 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:22.328300 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:22.328337 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:22.328524 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:22.328525 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:22.328526 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 23:24:22.328574 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:22.328575 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.329476 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.329477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:22.330052 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:22.330054 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.330552 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.330553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:22.330619 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:22.330633 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 23:24:22.330822 (+ 189us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 23:24:22.330824 (+ 2us) client_negotiation.cc:769] Sending connection context
0618 23:24:22.330861 (+ 37us) client_negotiation.cc:240] Negotiation successful
0618 23:24:22.330894 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260618 23:24:22.330971 10585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:22.328336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:45597 (local address 127.8.39.131:38833)
0618 23:24:22.328372 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:22.328374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:22.328382 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:24:22.328401 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:22.328402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:22.328410 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:22.328472 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:24:22.328653 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.329383 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.330140 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.330435 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.330509 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:22.330727 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:24:22.330760 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:24:22.330814 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:22.330864 (+ 50us) server_negotiation.cc:299] Negotiation successful
0618 23:24:22.330883 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260618 23:24:22.332931 10238 raft_consensus.cc:1261] T c2d11ab17fda469f90ebdda33a27911a P 1118c3f91fbe492cadf521766345707a [term 1 FOLLOWER]: Refusing update from remote peer 38ddfa85ddec470ca8a999e852c88dd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:24:22.333038 10377 raft_consensus.cc:1261] T c2d11ab17fda469f90ebdda33a27911a P 5a3086abd30a4331a257b50f39adf55e [term 1 FOLLOWER]: Refusing update from remote peer 38ddfa85ddec470ca8a999e852c88dd9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:24:22.333272 10595 consensus_queue.cc:1035] T c2d11ab17fda469f90ebdda33a27911a P 38ddfa85ddec470ca8a999e852c88dd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:22.333842 10597 consensus_queue.cc:1035] T c2d11ab17fda469f90ebdda33a27911a P 38ddfa85ddec470ca8a999e852c88dd9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:22.335189 10616 mvcc.cc:205] Tried to move back new op lower bound from 7298355455314542592 to 7298355454043553792. Current Snapshot: MvccSnapshot[applied={T|T < 7298355455314542592}]
I20260618 23:24:22.348289 10081 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:24:22.348407 10081 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:24:22.349327 10081 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7b49d74648e4730ba179de0645341e5: Sending DeleteTablet for 3 replicas of tablet c2d11ab17fda469f90ebdda33a27911a
I20260618 23:24:22.349606 10498 tablet_service.cc:1514] Processing DeleteTablet for tablet c2d11ab17fda469f90ebdda33a27911a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:22 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:48354
I20260618 23:24:22.349680 10218 tablet_service.cc:1514] Processing DeleteTablet for tablet c2d11ab17fda469f90ebdda33a27911a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:22 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:56268
I20260618 23:24:22.349864 10627 tablet_replica.cc:330] T c2d11ab17fda469f90ebdda33a27911a P 38ddfa85ddec470ca8a999e852c88dd9: stopping tablet replica
I20260618 23:24:22.349879 10358 tablet_service.cc:1514] Processing DeleteTablet for tablet c2d11ab17fda469f90ebdda33a27911a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:22 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:35636
I20260618 23:24:22.349965 10627 raft_consensus.cc:2229] T c2d11ab17fda469f90ebdda33a27911a P 38ddfa85ddec470ca8a999e852c88dd9 [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:24:22.350062 10627 raft_consensus.cc:2258] T c2d11ab17fda469f90ebdda33a27911a P 38ddfa85ddec470ca8a999e852c88dd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:22.350076 10628 tablet_replica.cc:330] T c2d11ab17fda469f90ebdda33a27911a P 5a3086abd30a4331a257b50f39adf55e: stopping tablet replica
I20260618 23:24:22.350160 10628 raft_consensus.cc:2229] T c2d11ab17fda469f90ebdda33a27911a P 5a3086abd30a4331a257b50f39adf55e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:24:22.350237 10628 raft_consensus.cc:2258] T c2d11ab17fda469f90ebdda33a27911a P 5a3086abd30a4331a257b50f39adf55e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:22.350561 10627 ts_tablet_manager.cc:1905] T c2d11ab17fda469f90ebdda33a27911a P 38ddfa85ddec470ca8a999e852c88dd9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:22.350589 10628 ts_tablet_manager.cc:1905] T c2d11ab17fda469f90ebdda33a27911a P 5a3086abd30a4331a257b50f39adf55e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:22.350893 10081 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37388:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:24:22.351035 10081 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:24:22.352245 10498 tablet_service.cc:1467] Processing CreateTablet for tablet 1a1616d2c0694dac883d422cecd2b069 (DEFAULT_TABLE table=test-table [id=0e57396868ce4887879a0336f2386455]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:22.352407 10628 ts_tablet_manager.cc:1918] T c2d11ab17fda469f90ebdda33a27911a P 5a3086abd30a4331a257b50f39adf55e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 23:24:22.352439 10498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a1616d2c0694dac883d422cecd2b069. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:22.352475 10628 log.cc:1196] T c2d11ab17fda469f90ebdda33a27911a P 5a3086abd30a4331a257b50f39adf55e: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-1/wal/wals/c2d11ab17fda469f90ebdda33a27911a
I20260618 23:24:22.352540 10218 tablet_service.cc:1467] Processing CreateTablet for tablet 1a1616d2c0694dac883d422cecd2b069 (DEFAULT_TABLE table=test-table [id=0e57396868ce4887879a0336f2386455]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:22.352654 10218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a1616d2c0694dac883d422cecd2b069. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:22.352737 10628 ts_tablet_manager.cc:1939] T c2d11ab17fda469f90ebdda33a27911a P 5a3086abd30a4331a257b50f39adf55e: Deleting consensus metadata
I20260618 23:24:22.353843 10629 tablet_replica.cc:330] T c2d11ab17fda469f90ebdda33a27911a P 1118c3f91fbe492cadf521766345707a: stopping tablet replica
I20260618 23:24:22.353891 10591 tablet_bootstrap.cc:492] T 1a1616d2c0694dac883d422cecd2b069 P 38ddfa85ddec470ca8a999e852c88dd9: Bootstrap starting.
I20260618 23:24:22.354053 10629 raft_consensus.cc:2229] T c2d11ab17fda469f90ebdda33a27911a P 1118c3f91fbe492cadf521766345707a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:24:22.354175 10629 raft_consensus.cc:2258] T c2d11ab17fda469f90ebdda33a27911a P 1118c3f91fbe492cadf521766345707a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:22.354305 10591 tablet_bootstrap.cc:654] T 1a1616d2c0694dac883d422cecd2b069 P 38ddfa85ddec470ca8a999e852c88dd9: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:22.354212 10587 tablet_bootstrap.cc:492] T 1a1616d2c0694dac883d422cecd2b069 P 1118c3f91fbe492cadf521766345707a: Bootstrap starting.
I20260618 23:24:22.354648 10066 catalog_manager.cc:4841] TS 5a3086abd30a4331a257b50f39adf55e (127.8.39.130:37285): tablet c2d11ab17fda469f90ebdda33a27911a (table test-table [id=3d09a097908247ef914e50452b2443ac]) successfully deleted
I20260618 23:24:22.354785 10587 tablet_bootstrap.cc:654] T 1a1616d2c0694dac883d422cecd2b069 P 1118c3f91fbe492cadf521766345707a: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:22.354790 10358 tablet_service.cc:1467] Processing CreateTablet for tablet 1a1616d2c0694dac883d422cecd2b069 (DEFAULT_TABLE table=test-table [id=0e57396868ce4887879a0336f2386455]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:22.354882 10358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a1616d2c0694dac883d422cecd2b069. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:22.355367 10587 tablet_bootstrap.cc:492] T 1a1616d2c0694dac883d422cecd2b069 P 1118c3f91fbe492cadf521766345707a: No bootstrap required, opened a new log
I20260618 23:24:22.355422 10587 ts_tablet_manager.cc:1397] T 1a1616d2c0694dac883d422cecd2b069 P 1118c3f91fbe492cadf521766345707a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:22.355569 10587 raft_consensus.cc:348] T 1a1616d2c0694dac883d422cecd2b069 P 1118c3f91fbe492cadf521766345707a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:22.355659 10587 raft_consensus.cc:374] T 1a1616d2c0694dac883d422cecd2b069 P 1118c3f91fbe492cadf521766345707a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:22.355682 10587 raft_consensus.cc:729] T 1a1616d2c0694dac883d422cecd2b069 P 1118c3f91fbe492cadf521766345707a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1118c3f91fbe492cadf521766345707a, State: Initialized, Role: FOLLOWER
I20260618 23:24:22.355744 10587 consensus_queue.cc:260] T 1a1616d2c0694dac883d422cecd2b069 P 1118c3f91fbe492cadf521766345707a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:22.355851 10587 ts_tablet_manager.cc:1428] T 1a1616d2c0694dac883d422cecd2b069 P 1118c3f91fbe492cadf521766345707a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:22.356307 10588 tablet_bootstrap.cc:492] T 1a1616d2c0694dac883d422cecd2b069 P 5a3086abd30a4331a257b50f39adf55e: Bootstrap starting.
I20260618 23:24:22.356374 10629 ts_tablet_manager.cc:1905] T c2d11ab17fda469f90ebdda33a27911a P 1118c3f91fbe492cadf521766345707a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:22.356662 10588 tablet_bootstrap.cc:654] T 1a1616d2c0694dac883d422cecd2b069 P 5a3086abd30a4331a257b50f39adf55e: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:22.357316 10588 tablet_bootstrap.cc:492] T 1a1616d2c0694dac883d422cecd2b069 P 5a3086abd30a4331a257b50f39adf55e: No bootstrap required, opened a new log
I20260618 23:24:22.357365 10588 ts_tablet_manager.cc:1397] T 1a1616d2c0694dac883d422cecd2b069 P 5a3086abd30a4331a257b50f39adf55e: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260618 23:24:22.357530 10588 raft_consensus.cc:348] T 1a1616d2c0694dac883d422cecd2b069 P 5a3086abd30a4331a257b50f39adf55e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:22.357591 10588 raft_consensus.cc:374] T 1a1616d2c0694dac883d422cecd2b069 P 5a3086abd30a4331a257b50f39adf55e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:22.357611 10588 raft_consensus.cc:729] T 1a1616d2c0694dac883d422cecd2b069 P 5a3086abd30a4331a257b50f39adf55e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a3086abd30a4331a257b50f39adf55e, State: Initialized, Role: FOLLOWER
I20260618 23:24:22.357650 10588 consensus_queue.cc:260] T 1a1616d2c0694dac883d422cecd2b069 P 5a3086abd30a4331a257b50f39adf55e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:22.357730 10588 ts_tablet_manager.cc:1428] T 1a1616d2c0694dac883d422cecd2b069 P 5a3086abd30a4331a257b50f39adf55e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:22.358055 10591 tablet_bootstrap.cc:492] T 1a1616d2c0694dac883d422cecd2b069 P 38ddfa85ddec470ca8a999e852c88dd9: No bootstrap required, opened a new log
I20260618 23:24:22.358108 10591 ts_tablet_manager.cc:1397] T 1a1616d2c0694dac883d422cecd2b069 P 38ddfa85ddec470ca8a999e852c88dd9: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260618 23:24:22.358300 10629 ts_tablet_manager.cc:1918] T c2d11ab17fda469f90ebdda33a27911a P 1118c3f91fbe492cadf521766345707a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 23:24:22.358301 10591 raft_consensus.cc:348] T 1a1616d2c0694dac883d422cecd2b069 P 38ddfa85ddec470ca8a999e852c88dd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:22.358366 10629 log.cc:1196] T c2d11ab17fda469f90ebdda33a27911a P 1118c3f91fbe492cadf521766345707a: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-0/wal/wals/c2d11ab17fda469f90ebdda33a27911a
I20260618 23:24:22.358376 10591 raft_consensus.cc:374] T 1a1616d2c0694dac883d422cecd2b069 P 38ddfa85ddec470ca8a999e852c88dd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:22.358402 10591 raft_consensus.cc:729] T 1a1616d2c0694dac883d422cecd2b069 P 38ddfa85ddec470ca8a999e852c88dd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ddfa85ddec470ca8a999e852c88dd9, State: Initialized, Role: FOLLOWER
I20260618 23:24:22.358454 10591 consensus_queue.cc:260] T 1a1616d2c0694dac883d422cecd2b069 P 38ddfa85ddec470ca8a999e852c88dd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:22.358572 10591 ts_tablet_manager.cc:1428] T 1a1616d2c0694dac883d422cecd2b069 P 38ddfa85ddec470ca8a999e852c88dd9: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260618 23:24:22.358635 10629 ts_tablet_manager.cc:1939] T c2d11ab17fda469f90ebdda33a27911a P 1118c3f91fbe492cadf521766345707a: Deleting consensus metadata
I20260618 23:24:22.358713 10627 ts_tablet_manager.cc:1918] T c2d11ab17fda469f90ebdda33a27911a P 38ddfa85ddec470ca8a999e852c88dd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 23:24:22.358769 10627 log.cc:1196] T c2d11ab17fda469f90ebdda33a27911a P 38ddfa85ddec470ca8a999e852c88dd9: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-2/wal/wals/c2d11ab17fda469f90ebdda33a27911a
I20260618 23:24:22.359054 10067 catalog_manager.cc:4841] TS 1118c3f91fbe492cadf521766345707a (127.8.39.129:41901): tablet c2d11ab17fda469f90ebdda33a27911a (table test-table [id=3d09a097908247ef914e50452b2443ac]) successfully deleted
I20260618 23:24:22.359162 10627 ts_tablet_manager.cc:1939] T c2d11ab17fda469f90ebdda33a27911a P 38ddfa85ddec470ca8a999e852c88dd9: Deleting consensus metadata
I20260618 23:24:22.359494 10067 catalog_manager.cc:4841] TS 38ddfa85ddec470ca8a999e852c88dd9 (127.8.39.131:38833): tablet c2d11ab17fda469f90ebdda33a27911a (table test-table [id=3d09a097908247ef914e50452b2443ac]) successfully deleted
I20260618 23:24:22.560770 10592 raft_consensus.cc:482] T 1a1616d2c0694dac883d422cecd2b069 P 1118c3f91fbe492cadf521766345707a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:24:22.560875 10592 raft_consensus.cc:504] T 1a1616d2c0694dac883d422cecd2b069 P 1118c3f91fbe492cadf521766345707a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:22.561259 10592 leader_election.cc:288] T 1a1616d2c0694dac883d422cecd2b069 P 1118c3f91fbe492cadf521766345707a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a3086abd30a4331a257b50f39adf55e (127.8.39.130:37285), 38ddfa85ddec470ca8a999e852c88dd9 (127.8.39.131:38833)
I20260618 23:24:22.564332 10634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:22.561444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:38833 (local address 127.8.39.129:44837)
0618 23:24:22.561575 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:22.561585 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:22.561646 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:22.561973 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:22.561974 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:22.561975 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:22.562013 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:22.562015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.562641 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.562642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:22.563436 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:22.563438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.564062 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.564063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:22.564124 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:22.564128 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:24:22.564209 (+ 81us) client_negotiation.cc:240] Negotiation successful
0618 23:24:22.564255 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":58,"threads_started":1}
I20260618 23:24:22.564618 10585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:22.561511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:44837 (local address 127.8.39.131:38833)
0618 23:24:22.561567 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:22.561570 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:22.561631 (+ 61us) server_negotiation.cc:407] Connection header received
0618 23:24:22.561768 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:22.561769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:22.561780 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:22.561957 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:22.562091 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.562560 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.563554 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.563983 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.564047 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:22.564055 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:22.564325 (+ 270us) server_negotiation.cc:299] Negotiation successful
0618 23:24:22.564428 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260618 23:24:22.564757 10635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:22.561593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:52683 (local address 127.8.39.130:37285)
0618 23:24:22.561711 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:22.561713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:22.561721 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:24:22.561749 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:22.561750 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:22.561761 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:22.561816 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:22.562106 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.562591 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.563720 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.564144 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.564211 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:22.564219 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:22.564609 (+ 390us) server_negotiation.cc:299] Negotiation successful
0618 23:24:22.564658 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
I20260618 23:24:22.564777 10633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:22.561432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:37285 (local address 127.8.39.129:52683)
0618 23:24:22.561564 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:22.561576 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:22.561634 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:22.561941 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:22.561942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:22.561944 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:22.562005 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:22.562008 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.562700 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.562701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:22.563622 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:22.563624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.564333 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.564336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:22.564435 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:22.564442 (+ 7us) client_negotiation.cc:769] Sending connection context
0618 23:24:22.564606 (+ 164us) client_negotiation.cc:240] Negotiation successful
0618 23:24:22.564687 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":101,"thread_start_us":72,"threads_started":1}
I20260618 23:24:22.564822 10501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1616d2c0694dac883d422cecd2b069" candidate_uuid: "1118c3f91fbe492cadf521766345707a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ddfa85ddec470ca8a999e852c88dd9" is_pre_election: true
I20260618 23:24:22.564952 10501 raft_consensus.cc:2454] T 1a1616d2c0694dac883d422cecd2b069 P 38ddfa85ddec470ca8a999e852c88dd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1118c3f91fbe492cadf521766345707a in term 0.
I20260618 23:24:22.565037 10377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1616d2c0694dac883d422cecd2b069" candidate_uuid: "1118c3f91fbe492cadf521766345707a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a3086abd30a4331a257b50f39adf55e" is_pre_election: true
I20260618 23:24:22.565153 10377 raft_consensus.cc:2454] T 1a1616d2c0694dac883d422cecd2b069 P 5a3086abd30a4331a257b50f39adf55e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1118c3f91fbe492cadf521766345707a in term 0.
I20260618 23:24:22.565168 10171 leader_election.cc:302] T 1a1616d2c0694dac883d422cecd2b069 P 1118c3f91fbe492cadf521766345707a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1118c3f91fbe492cadf521766345707a, 38ddfa85ddec470ca8a999e852c88dd9; no voters:
I20260618 23:24:22.565306 10592 raft_consensus.cc:2781] T 1a1616d2c0694dac883d422cecd2b069 P 1118c3f91fbe492cadf521766345707a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:24:22.565353 10592 raft_consensus.cc:482] T 1a1616d2c0694dac883d422cecd2b069 P 1118c3f91fbe492cadf521766345707a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:24:22.565374 10592 raft_consensus.cc:3037] T 1a1616d2c0694dac883d422cecd2b069 P 1118c3f91fbe492cadf521766345707a [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:22.566038 10592 raft_consensus.cc:504] T 1a1616d2c0694dac883d422cecd2b069 P 1118c3f91fbe492cadf521766345707a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:22.566186 10592 leader_election.cc:288] T 1a1616d2c0694dac883d422cecd2b069 P 1118c3f91fbe492cadf521766345707a [CANDIDATE]: Term 1 election: Requested vote from peers 5a3086abd30a4331a257b50f39adf55e (127.8.39.130:37285), 38ddfa85ddec470ca8a999e852c88dd9 (127.8.39.131:38833)
I20260618 23:24:22.566329 10377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1616d2c0694dac883d422cecd2b069" candidate_uuid: "1118c3f91fbe492cadf521766345707a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a3086abd30a4331a257b50f39adf55e"
I20260618 23:24:22.566376 10501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a1616d2c0694dac883d422cecd2b069" candidate_uuid: "1118c3f91fbe492cadf521766345707a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ddfa85ddec470ca8a999e852c88dd9"
I20260618 23:24:22.566422 10377 raft_consensus.cc:3037] T 1a1616d2c0694dac883d422cecd2b069 P 5a3086abd30a4331a257b50f39adf55e [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:22.566426 10501 raft_consensus.cc:3037] T 1a1616d2c0694dac883d422cecd2b069 P 38ddfa85ddec470ca8a999e852c88dd9 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:22.567013 10377 raft_consensus.cc:2454] T 1a1616d2c0694dac883d422cecd2b069 P 5a3086abd30a4331a257b50f39adf55e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1118c3f91fbe492cadf521766345707a in term 1.
I20260618 23:24:22.567054 10501 raft_consensus.cc:2454] T 1a1616d2c0694dac883d422cecd2b069 P 38ddfa85ddec470ca8a999e852c88dd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1118c3f91fbe492cadf521766345707a in term 1.
I20260618 23:24:22.567203 10171 leader_election.cc:302] T 1a1616d2c0694dac883d422cecd2b069 P 1118c3f91fbe492cadf521766345707a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1118c3f91fbe492cadf521766345707a, 38ddfa85ddec470ca8a999e852c88dd9; no voters:
I20260618 23:24:22.567307 10592 raft_consensus.cc:2781] T 1a1616d2c0694dac883d422cecd2b069 P 1118c3f91fbe492cadf521766345707a [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:22.567437 10592 raft_consensus.cc:686] T 1a1616d2c0694dac883d422cecd2b069 P 1118c3f91fbe492cadf521766345707a [term 1 LEADER]: Becoming Leader. State: Replica: 1118c3f91fbe492cadf521766345707a, State: Running, Role: LEADER
I20260618 23:24:22.567520 10592 consensus_queue.cc:237] T 1a1616d2c0694dac883d422cecd2b069 P 1118c3f91fbe492cadf521766345707a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:22.568218 10081 catalog_manager.cc:5495] T 1a1616d2c0694dac883d422cecd2b069 P 1118c3f91fbe492cadf521766345707a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1118c3f91fbe492cadf521766345707a (127.8.39.129). New cstate: current_term: 1 leader_uuid: "1118c3f91fbe492cadf521766345707a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } health_report { overall_health: UNKNOWN } } }
I20260618 23:24:22.591738 10638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:22.589204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56274 (local address 127.8.39.129:41901)
0618 23:24:22.589345 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:22.589346 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:22.589353 (+ 7us) server_negotiation.cc:407] Connection header received
0618 23:24:22.589390 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:22.589391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:22.589416 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:22.589501 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:24:22.589641 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.590144 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.590873 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.591117 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.591197 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:22.591357 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:24:22.591476 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:24:22.591563 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:22.591616 (+ 53us) server_negotiation.cc:299] Negotiation successful
0618 23:24:22.591638 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
I20260618 23:24:22.596312 10585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:22.594215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:34101 (local address 127.8.39.131:38833)
0618 23:24:22.594276 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:22.594277 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:22.594401 (+ 124us) server_negotiation.cc:407] Connection header received
0618 23:24:22.594439 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:22.594440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:22.594447 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:22.594496 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:24:22.594579 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.595012 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.595710 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.595913 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.595967 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:22.596139 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:24:22.596162 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:24:22.596198 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:22.596247 (+ 49us) server_negotiation.cc:299] Negotiation successful
0618 23:24:22.596260 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260618 23:24:22.596361 10641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:22.594141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:38833 (local address 127.8.39.129:34101)
0618 23:24:22.594341 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:22.594350 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:22.594399 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:22.594499 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:22.594500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:22.594500 (+ 0us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 23:24:22.594542 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:22.594543 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.595064 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.595065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:22.595630 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:22.595631 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.595992 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.595993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:22.596044 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:22.596055 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 23:24:22.596211 (+ 156us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 23:24:22.596213 (+ 2us) client_negotiation.cc:769] Sending connection context
0618 23:24:22.596246 (+ 33us) client_negotiation.cc:240] Negotiation successful
0618 23:24:22.596288 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":91,"threads_started":1}
I20260618 23:24:22.598562 10501 raft_consensus.cc:1261] T 1a1616d2c0694dac883d422cecd2b069 P 38ddfa85ddec470ca8a999e852c88dd9 [term 1 FOLLOWER]: Refusing update from remote peer 1118c3f91fbe492cadf521766345707a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:24:22.598555 10377 raft_consensus.cc:1261] T 1a1616d2c0694dac883d422cecd2b069 P 5a3086abd30a4331a257b50f39adf55e [term 1 FOLLOWER]: Refusing update from remote peer 1118c3f91fbe492cadf521766345707a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:24:22.598894 10636 consensus_queue.cc:1035] T 1a1616d2c0694dac883d422cecd2b069 P 1118c3f91fbe492cadf521766345707a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:22.599043 10592 consensus_queue.cc:1035] T 1a1616d2c0694dac883d422cecd2b069 P 1118c3f91fbe492cadf521766345707a [LEADER]: Connected to new peer: Peer: permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:22.600028 10613 mvcc.cc:205] Tried to move back new op lower bound from 7298355456400998400 to 7298355456277192704. Current Snapshot: MvccSnapshot[applied={T|T < 7298355456400998400}]
I20260618 23:24:22.606283 10570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:22.603952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:41901 (local address 127.8.39.131:39663)
0618 23:24:22.603991 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:22.604001 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:22.604037 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:22.604300 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:22.604301 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:22.604302 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 23:24:22.604343 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:22.604344 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.604957 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.604958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:22.605508 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:22.605510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.605884 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.605885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:22.605944 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:22.605949 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 23:24:22.606127 (+ 178us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 23:24:22.606130 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:24:22.606177 (+ 47us) client_negotiation.cc:240] Negotiation successful
0618 23:24:22.606219 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260618 23:24:22.606272 10638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:22.604098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:39663 (local address 127.8.39.129:41901)
0618 23:24:22.604168 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:22.604170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:22.604177 (+ 7us) server_negotiation.cc:407] Connection header received
0618 23:24:22.604198 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:22.604198 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:22.604207 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:22.604294 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:24:22.604394 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.604862 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.605595 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:22.605806 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:22.605886 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:22.606024 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:24:22.606054 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:24:22.606126 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:22.606180 (+ 54us) server_negotiation.cc:299] Negotiation successful
0618 23:24:22.606198 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260618 23:24:22.606693 10613 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260618 23:24:24.131413 10080 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:24:24.131537 10080 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37388:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:24:24.132752 10080 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7b49d74648e4730ba179de0645341e5: Sending DeleteTablet for 3 replicas of tablet 1a1616d2c0694dac883d422cecd2b069
I20260618 23:24:24.133102 10218 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a1616d2c0694dac883d422cecd2b069 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:24 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:56268
I20260618 23:24:24.133168 10498 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a1616d2c0694dac883d422cecd2b069 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:24 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:48354
I20260618 23:24:24.133328 10358 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a1616d2c0694dac883d422cecd2b069 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:24 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:35636
I20260618 23:24:24.133527 10669 tablet_replica.cc:330] T 1a1616d2c0694dac883d422cecd2b069 P 5a3086abd30a4331a257b50f39adf55e: stopping tablet replica
I20260618 23:24:24.133625 10669 raft_consensus.cc:2229] T 1a1616d2c0694dac883d422cecd2b069 P 5a3086abd30a4331a257b50f39adf55e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:24:24.133714 10669 raft_consensus.cc:2258] T 1a1616d2c0694dac883d422cecd2b069 P 5a3086abd30a4331a257b50f39adf55e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:24.134791 10667 tablet_replica.cc:330] T 1a1616d2c0694dac883d422cecd2b069 P 1118c3f91fbe492cadf521766345707a: stopping tablet replica
I20260618 23:24:24.134939 10668 tablet_replica.cc:330] T 1a1616d2c0694dac883d422cecd2b069 P 38ddfa85ddec470ca8a999e852c88dd9: stopping tablet replica
I20260618 23:24:24.135028 10668 raft_consensus.cc:2229] T 1a1616d2c0694dac883d422cecd2b069 P 38ddfa85ddec470ca8a999e852c88dd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:24:24.135126 10667 raft_consensus.cc:2229] T 1a1616d2c0694dac883d422cecd2b069 P 1118c3f91fbe492cadf521766345707a [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:24:24.135205 10668 raft_consensus.cc:2258] T 1a1616d2c0694dac883d422cecd2b069 P 38ddfa85ddec470ca8a999e852c88dd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:24.135332 10667 raft_consensus.cc:2258] T 1a1616d2c0694dac883d422cecd2b069 P 1118c3f91fbe492cadf521766345707a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:24.135655 10667 ts_tablet_manager.cc:1905] T 1a1616d2c0694dac883d422cecd2b069 P 1118c3f91fbe492cadf521766345707a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:24.135887 10668 ts_tablet_manager.cc:1905] T 1a1616d2c0694dac883d422cecd2b069 P 38ddfa85ddec470ca8a999e852c88dd9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:24.137321 10669 ts_tablet_manager.cc:1905] T 1a1616d2c0694dac883d422cecd2b069 P 5a3086abd30a4331a257b50f39adf55e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:24.137372 10668 ts_tablet_manager.cc:1918] T 1a1616d2c0694dac883d422cecd2b069 P 38ddfa85ddec470ca8a999e852c88dd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260618 23:24:24.137379 10667 ts_tablet_manager.cc:1918] T 1a1616d2c0694dac883d422cecd2b069 P 1118c3f91fbe492cadf521766345707a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260618 23:24:24.137423 10668 log.cc:1196] T 1a1616d2c0694dac883d422cecd2b069 P 38ddfa85ddec470ca8a999e852c88dd9: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-2/wal/wals/1a1616d2c0694dac883d422cecd2b069
I20260618 23:24:24.137423 10667 log.cc:1196] T 1a1616d2c0694dac883d422cecd2b069 P 1118c3f91fbe492cadf521766345707a: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-0/wal/wals/1a1616d2c0694dac883d422cecd2b069
I20260618 23:24:24.137686 10667 ts_tablet_manager.cc:1939] T 1a1616d2c0694dac883d422cecd2b069 P 1118c3f91fbe492cadf521766345707a: Deleting consensus metadata
I20260618 23:24:24.137696 10668 ts_tablet_manager.cc:1939] T 1a1616d2c0694dac883d422cecd2b069 P 38ddfa85ddec470ca8a999e852c88dd9: Deleting consensus metadata
I20260618 23:24:24.138181 10067 catalog_manager.cc:4841] TS 38ddfa85ddec470ca8a999e852c88dd9 (127.8.39.131:38833): tablet 1a1616d2c0694dac883d422cecd2b069 (table test-table [id=0e57396868ce4887879a0336f2386455]) successfully deleted
I20260618 23:24:24.138264 10067 catalog_manager.cc:4841] TS 1118c3f91fbe492cadf521766345707a (127.8.39.129:41901): tablet 1a1616d2c0694dac883d422cecd2b069 (table test-table [id=0e57396868ce4887879a0336f2386455]) successfully deleted
I20260618 23:24:24.138782 10669 ts_tablet_manager.cc:1918] T 1a1616d2c0694dac883d422cecd2b069 P 5a3086abd30a4331a257b50f39adf55e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260618 23:24:24.138861 10669 log.cc:1196] T 1a1616d2c0694dac883d422cecd2b069 P 5a3086abd30a4331a257b50f39adf55e: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-1/wal/wals/1a1616d2c0694dac883d422cecd2b069
I20260618 23:24:24.139066 10669 ts_tablet_manager.cc:1939] T 1a1616d2c0694dac883d422cecd2b069 P 5a3086abd30a4331a257b50f39adf55e: Deleting consensus metadata
I20260618 23:24:24.139443 10066 catalog_manager.cc:4841] TS 5a3086abd30a4331a257b50f39adf55e (127.8.39.130:37285): tablet 1a1616d2c0694dac883d422cecd2b069 (table test-table [id=0e57396868ce4887879a0336f2386455]) successfully deleted
Jun 18 23:24:24 dist-test-slave-47qt krb5kdc[10036](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825064, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 18 23:24:24 dist-test-slave-47qt krb5kdc[10036](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825064, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:24.165086 10677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:24.152293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47056 (local address 127.8.39.190:33587)
0618 23:24:24.152446 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:24.152448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:24.152514 (+ 66us) server_negotiation.cc:407] Connection header received
0618 23:24:24.152628 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:24.152629 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:24.152640 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:24.152728 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:24.153376 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:24.153901 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:24.154654 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:24.154853 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:24.154938 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:24.162158 (+ 7220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:24.162160 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:24.162160 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:24.162169 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:24.163596 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:24.164087 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:24.164090 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:24.164090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:24.164230 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:24.164481 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:24.164482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:24.164482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:24.164605 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:24.164717 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:24.164839 (+ 122us) server_negotiation.cc:299] Negotiation successful
0618 23:24:24.164963 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20260618 23:24:24.167238 10080 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47056:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:24:24.167390 10080 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:24:24.168870 10218 tablet_service.cc:1467] Processing CreateTablet for tablet bbca4a2238e34b26833a326d105922d9 (DEFAULT_TABLE table=test-table [id=9488d99086aa4e11abf008cf64d1990d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:24.168870 10358 tablet_service.cc:1467] Processing CreateTablet for tablet bbca4a2238e34b26833a326d105922d9 (DEFAULT_TABLE table=test-table [id=9488d99086aa4e11abf008cf64d1990d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:24.168881 10498 tablet_service.cc:1467] Processing CreateTablet for tablet bbca4a2238e34b26833a326d105922d9 (DEFAULT_TABLE table=test-table [id=9488d99086aa4e11abf008cf64d1990d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:24.169039 10498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbca4a2238e34b26833a326d105922d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:24.169181 10218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbca4a2238e34b26833a326d105922d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:24.169183 10358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbca4a2238e34b26833a326d105922d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:24.170190 10678 tablet_bootstrap.cc:492] T bbca4a2238e34b26833a326d105922d9 P 38ddfa85ddec470ca8a999e852c88dd9: Bootstrap starting.
I20260618 23:24:24.170702 10678 tablet_bootstrap.cc:654] T bbca4a2238e34b26833a326d105922d9 P 38ddfa85ddec470ca8a999e852c88dd9: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:24.170709 10679 tablet_bootstrap.cc:492] T bbca4a2238e34b26833a326d105922d9 P 1118c3f91fbe492cadf521766345707a: Bootstrap starting.
I20260618 23:24:24.170859 10680 tablet_bootstrap.cc:492] T bbca4a2238e34b26833a326d105922d9 P 5a3086abd30a4331a257b50f39adf55e: Bootstrap starting.
I20260618 23:24:24.171037 10679 tablet_bootstrap.cc:654] T bbca4a2238e34b26833a326d105922d9 P 1118c3f91fbe492cadf521766345707a: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:24.171269 10680 tablet_bootstrap.cc:654] T bbca4a2238e34b26833a326d105922d9 P 5a3086abd30a4331a257b50f39adf55e: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:24.171416 10678 tablet_bootstrap.cc:492] T bbca4a2238e34b26833a326d105922d9 P 38ddfa85ddec470ca8a999e852c88dd9: No bootstrap required, opened a new log
I20260618 23:24:24.171484 10678 ts_tablet_manager.cc:1397] T bbca4a2238e34b26833a326d105922d9 P 38ddfa85ddec470ca8a999e852c88dd9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:24.171607 10679 tablet_bootstrap.cc:492] T bbca4a2238e34b26833a326d105922d9 P 1118c3f91fbe492cadf521766345707a: No bootstrap required, opened a new log
I20260618 23:24:24.171650 10679 ts_tablet_manager.cc:1397] T bbca4a2238e34b26833a326d105922d9 P 1118c3f91fbe492cadf521766345707a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:24.171598 10678 raft_consensus.cc:348] T bbca4a2238e34b26833a326d105922d9 P 38ddfa85ddec470ca8a999e852c88dd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:24.171697 10678 raft_consensus.cc:374] T bbca4a2238e34b26833a326d105922d9 P 38ddfa85ddec470ca8a999e852c88dd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:24.171720 10678 raft_consensus.cc:729] T bbca4a2238e34b26833a326d105922d9 P 38ddfa85ddec470ca8a999e852c88dd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ddfa85ddec470ca8a999e852c88dd9, State: Initialized, Role: FOLLOWER
I20260618 23:24:24.171763 10678 consensus_queue.cc:260] T bbca4a2238e34b26833a326d105922d9 P 38ddfa85ddec470ca8a999e852c88dd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:24.171787 10679 raft_consensus.cc:348] T bbca4a2238e34b26833a326d105922d9 P 1118c3f91fbe492cadf521766345707a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:24.171905 10678 ts_tablet_manager.cc:1428] T bbca4a2238e34b26833a326d105922d9 P 38ddfa85ddec470ca8a999e852c88dd9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:24.171938 10679 raft_consensus.cc:374] T bbca4a2238e34b26833a326d105922d9 P 1118c3f91fbe492cadf521766345707a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:24.172025 10679 raft_consensus.cc:729] T bbca4a2238e34b26833a326d105922d9 P 1118c3f91fbe492cadf521766345707a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1118c3f91fbe492cadf521766345707a, State: Initialized, Role: FOLLOWER
I20260618 23:24:24.172101 10680 tablet_bootstrap.cc:492] T bbca4a2238e34b26833a326d105922d9 P 5a3086abd30a4331a257b50f39adf55e: No bootstrap required, opened a new log
I20260618 23:24:24.172151 10680 ts_tablet_manager.cc:1397] T bbca4a2238e34b26833a326d105922d9 P 5a3086abd30a4331a257b50f39adf55e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:24.172125 10679 consensus_queue.cc:260] T bbca4a2238e34b26833a326d105922d9 P 1118c3f91fbe492cadf521766345707a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:24.172252 10680 raft_consensus.cc:348] T bbca4a2238e34b26833a326d105922d9 P 5a3086abd30a4331a257b50f39adf55e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:24.172314 10680 raft_consensus.cc:374] T bbca4a2238e34b26833a326d105922d9 P 5a3086abd30a4331a257b50f39adf55e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:24.172333 10680 raft_consensus.cc:729] T bbca4a2238e34b26833a326d105922d9 P 5a3086abd30a4331a257b50f39adf55e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a3086abd30a4331a257b50f39adf55e, State: Initialized, Role: FOLLOWER
I20260618 23:24:24.172346 10679 ts_tablet_manager.cc:1428] T bbca4a2238e34b26833a326d105922d9 P 1118c3f91fbe492cadf521766345707a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:24.172374 10680 consensus_queue.cc:260] T bbca4a2238e34b26833a326d105922d9 P 5a3086abd30a4331a257b50f39adf55e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:24.172544 10680 ts_tablet_manager.cc:1428] T bbca4a2238e34b26833a326d105922d9 P 5a3086abd30a4331a257b50f39adf55e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:24.248283 10661 raft_consensus.cc:482] T bbca4a2238e34b26833a326d105922d9 P 1118c3f91fbe492cadf521766345707a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:24:24.248394 10661 raft_consensus.cc:504] T bbca4a2238e34b26833a326d105922d9 P 1118c3f91fbe492cadf521766345707a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:24.248600 10661 leader_election.cc:288] T bbca4a2238e34b26833a326d105922d9 P 1118c3f91fbe492cadf521766345707a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5a3086abd30a4331a257b50f39adf55e (127.8.39.130:37285), 38ddfa85ddec470ca8a999e852c88dd9 (127.8.39.131:38833)
I20260618 23:24:24.248823 10377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbca4a2238e34b26833a326d105922d9" candidate_uuid: "1118c3f91fbe492cadf521766345707a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a3086abd30a4331a257b50f39adf55e" is_pre_election: true
I20260618 23:24:24.248831 10501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbca4a2238e34b26833a326d105922d9" candidate_uuid: "1118c3f91fbe492cadf521766345707a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ddfa85ddec470ca8a999e852c88dd9" is_pre_election: true
I20260618 23:24:24.248948 10377 raft_consensus.cc:2454] T bbca4a2238e34b26833a326d105922d9 P 5a3086abd30a4331a257b50f39adf55e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1118c3f91fbe492cadf521766345707a in term 0.
I20260618 23:24:24.248971 10501 raft_consensus.cc:2454] T bbca4a2238e34b26833a326d105922d9 P 38ddfa85ddec470ca8a999e852c88dd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1118c3f91fbe492cadf521766345707a in term 0.
I20260618 23:24:24.249101 10170 leader_election.cc:302] T bbca4a2238e34b26833a326d105922d9 P 1118c3f91fbe492cadf521766345707a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1118c3f91fbe492cadf521766345707a, 5a3086abd30a4331a257b50f39adf55e; no voters:
I20260618 23:24:24.249224 10661 raft_consensus.cc:2781] T bbca4a2238e34b26833a326d105922d9 P 1118c3f91fbe492cadf521766345707a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:24:24.249259 10661 raft_consensus.cc:482] T bbca4a2238e34b26833a326d105922d9 P 1118c3f91fbe492cadf521766345707a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:24:24.249279 10661 raft_consensus.cc:3037] T bbca4a2238e34b26833a326d105922d9 P 1118c3f91fbe492cadf521766345707a [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:24.249982 10661 raft_consensus.cc:504] T bbca4a2238e34b26833a326d105922d9 P 1118c3f91fbe492cadf521766345707a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:24.250118 10661 leader_election.cc:288] T bbca4a2238e34b26833a326d105922d9 P 1118c3f91fbe492cadf521766345707a [CANDIDATE]: Term 1 election: Requested vote from peers 5a3086abd30a4331a257b50f39adf55e (127.8.39.130:37285), 38ddfa85ddec470ca8a999e852c88dd9 (127.8.39.131:38833)
I20260618 23:24:24.250280 10377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbca4a2238e34b26833a326d105922d9" candidate_uuid: "1118c3f91fbe492cadf521766345707a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a3086abd30a4331a257b50f39adf55e"
I20260618 23:24:24.250320 10501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbca4a2238e34b26833a326d105922d9" candidate_uuid: "1118c3f91fbe492cadf521766345707a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ddfa85ddec470ca8a999e852c88dd9"
I20260618 23:24:24.250373 10377 raft_consensus.cc:3037] T bbca4a2238e34b26833a326d105922d9 P 5a3086abd30a4331a257b50f39adf55e [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:24.250398 10501 raft_consensus.cc:3037] T bbca4a2238e34b26833a326d105922d9 P 38ddfa85ddec470ca8a999e852c88dd9 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:24.251220 10501 raft_consensus.cc:2454] T bbca4a2238e34b26833a326d105922d9 P 38ddfa85ddec470ca8a999e852c88dd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1118c3f91fbe492cadf521766345707a in term 1.
I20260618 23:24:24.251220 10377 raft_consensus.cc:2454] T bbca4a2238e34b26833a326d105922d9 P 5a3086abd30a4331a257b50f39adf55e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1118c3f91fbe492cadf521766345707a in term 1.
I20260618 23:24:24.251407 10171 leader_election.cc:302] T bbca4a2238e34b26833a326d105922d9 P 1118c3f91fbe492cadf521766345707a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1118c3f91fbe492cadf521766345707a, 38ddfa85ddec470ca8a999e852c88dd9; no voters:
I20260618 23:24:24.251513 10661 raft_consensus.cc:2781] T bbca4a2238e34b26833a326d105922d9 P 1118c3f91fbe492cadf521766345707a [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:24.251578 10661 raft_consensus.cc:686] T bbca4a2238e34b26833a326d105922d9 P 1118c3f91fbe492cadf521766345707a [term 1 LEADER]: Becoming Leader. State: Replica: 1118c3f91fbe492cadf521766345707a, State: Running, Role: LEADER
I20260618 23:24:24.251643 10661 consensus_queue.cc:237] T bbca4a2238e34b26833a326d105922d9 P 1118c3f91fbe492cadf521766345707a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:24.252264 10079 catalog_manager.cc:5495] T bbca4a2238e34b26833a326d105922d9 P 1118c3f91fbe492cadf521766345707a reported cstate change: term changed from 0 to 1, leader changed from <none> to 1118c3f91fbe492cadf521766345707a (127.8.39.129). New cstate: current_term: 1 leader_uuid: "1118c3f91fbe492cadf521766345707a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } health_report { overall_health: UNKNOWN } } }
I20260618 23:24:24.269695 10687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:24.267307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55100 (local address 127.8.39.129:41901)
0618 23:24:24.267491 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:24.267494 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:24.267504 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:24:24.267535 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:24.267536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:24.267546 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:24.267622 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:24:24.267803 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:24.268341 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:24.269155 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:24.269341 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:24.269396 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:24.269515 (+ 119us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:24:24.269542 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:24:24.269572 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:24.269609 (+ 37us) server_negotiation.cc:299] Negotiation successful
0618 23:24:24.269624 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":125,"threads_started":1}
I20260618 23:24:24.271955 10501 raft_consensus.cc:1261] T bbca4a2238e34b26833a326d105922d9 P 38ddfa85ddec470ca8a999e852c88dd9 [term 1 FOLLOWER]: Refusing update from remote peer 1118c3f91fbe492cadf521766345707a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:24:24.271955 10377 raft_consensus.cc:1261] T bbca4a2238e34b26833a326d105922d9 P 5a3086abd30a4331a257b50f39adf55e [term 1 FOLLOWER]: Refusing update from remote peer 1118c3f91fbe492cadf521766345707a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:24:24.272149 10644 consensus_queue.cc:1035] T bbca4a2238e34b26833a326d105922d9 P 1118c3f91fbe492cadf521766345707a [LEADER]: Connected to new peer: Peer: permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:24.272286 10644 consensus_queue.cc:1035] T bbca4a2238e34b26833a326d105922d9 P 1118c3f91fbe492cadf521766345707a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:24.281642 10080 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:24:24.281728 10080 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:24:24.282374 10080 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7b49d74648e4730ba179de0645341e5: Sending DeleteTablet for 3 replicas of tablet bbca4a2238e34b26833a326d105922d9
I20260618 23:24:24.282653 10218 tablet_service.cc:1514] Processing DeleteTablet for tablet bbca4a2238e34b26833a326d105922d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:24 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:56268
I20260618 23:24:24.282692 10498 tablet_service.cc:1514] Processing DeleteTablet for tablet bbca4a2238e34b26833a326d105922d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:24 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:48354
I20260618 23:24:24.282760 10358 tablet_service.cc:1514] Processing DeleteTablet for tablet bbca4a2238e34b26833a326d105922d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:24 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:35636
I20260618 23:24:24.282853 10667 tablet_replica.cc:330] T bbca4a2238e34b26833a326d105922d9 P 1118c3f91fbe492cadf521766345707a: stopping tablet replica
I20260618 23:24:24.282856 10669 tablet_replica.cc:330] T bbca4a2238e34b26833a326d105922d9 P 5a3086abd30a4331a257b50f39adf55e: stopping tablet replica
I20260618 23:24:24.282902 10667 raft_consensus.cc:2229] T bbca4a2238e34b26833a326d105922d9 P 1118c3f91fbe492cadf521766345707a [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:24:24.282917 10669 raft_consensus.cc:2229] T bbca4a2238e34b26833a326d105922d9 P 5a3086abd30a4331a257b50f39adf55e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:24:24.282951 10667 raft_consensus.cc:2258] T bbca4a2238e34b26833a326d105922d9 P 1118c3f91fbe492cadf521766345707a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:24.282974 10669 raft_consensus.cc:2258] T bbca4a2238e34b26833a326d105922d9 P 5a3086abd30a4331a257b50f39adf55e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:24.282893 10080 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47056:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260618 23:24:24.283025 10668 tablet_replica.cc:330] T bbca4a2238e34b26833a326d105922d9 P 38ddfa85ddec470ca8a999e852c88dd9: stopping tablet replica
W20260618 23:24:24.283053 10080 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:24:24.283074 10668 raft_consensus.cc:2229] T bbca4a2238e34b26833a326d105922d9 P 38ddfa85ddec470ca8a999e852c88dd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:24:24.283113 10668 raft_consensus.cc:2258] T bbca4a2238e34b26833a326d105922d9 P 38ddfa85ddec470ca8a999e852c88dd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:24.283293 10669 ts_tablet_manager.cc:1905] T bbca4a2238e34b26833a326d105922d9 P 5a3086abd30a4331a257b50f39adf55e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:24.283538 10668 ts_tablet_manager.cc:1905] T bbca4a2238e34b26833a326d105922d9 P 38ddfa85ddec470ca8a999e852c88dd9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:24.284629 10218 tablet_service.cc:1467] Processing CreateTablet for tablet 0ae4a92fdcdd41a787a3cdf620212d38 (DEFAULT_TABLE table=test-table [id=5033b0ebf89e4b4bb351a7f963d06441]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:24.284739 10218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ae4a92fdcdd41a787a3cdf620212d38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:24.284766 10498 tablet_service.cc:1467] Processing CreateTablet for tablet 0ae4a92fdcdd41a787a3cdf620212d38 (DEFAULT_TABLE table=test-table [id=5033b0ebf89e4b4bb351a7f963d06441]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:24.284988 10669 ts_tablet_manager.cc:1918] T bbca4a2238e34b26833a326d105922d9 P 5a3086abd30a4331a257b50f39adf55e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 23:24:24.285030 10498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ae4a92fdcdd41a787a3cdf620212d38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:24.285053 10669 log.cc:1196] T bbca4a2238e34b26833a326d105922d9 P 5a3086abd30a4331a257b50f39adf55e: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-1/wal/wals/bbca4a2238e34b26833a326d105922d9
I20260618 23:24:24.285287 10358 tablet_service.cc:1467] Processing CreateTablet for tablet 0ae4a92fdcdd41a787a3cdf620212d38 (DEFAULT_TABLE table=test-table [id=5033b0ebf89e4b4bb351a7f963d06441]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:24.285321 10669 ts_tablet_manager.cc:1939] T bbca4a2238e34b26833a326d105922d9 P 5a3086abd30a4331a257b50f39adf55e: Deleting consensus metadata
I20260618 23:24:24.285386 10358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ae4a92fdcdd41a787a3cdf620212d38. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:24.286329 10679 tablet_bootstrap.cc:492] T 0ae4a92fdcdd41a787a3cdf620212d38 P 1118c3f91fbe492cadf521766345707a: Bootstrap starting.
I20260618 23:24:24.286476 10066 catalog_manager.cc:4841] TS 5a3086abd30a4331a257b50f39adf55e (127.8.39.130:37285): tablet bbca4a2238e34b26833a326d105922d9 (table test-table [id=9488d99086aa4e11abf008cf64d1990d]) successfully deleted
I20260618 23:24:24.286559 10680 tablet_bootstrap.cc:492] T 0ae4a92fdcdd41a787a3cdf620212d38 P 5a3086abd30a4331a257b50f39adf55e: Bootstrap starting.
I20260618 23:24:24.286569 10678 tablet_bootstrap.cc:492] T 0ae4a92fdcdd41a787a3cdf620212d38 P 38ddfa85ddec470ca8a999e852c88dd9: Bootstrap starting.
I20260618 23:24:24.286662 10679 tablet_bootstrap.cc:654] T 0ae4a92fdcdd41a787a3cdf620212d38 P 1118c3f91fbe492cadf521766345707a: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:24.286903 10678 tablet_bootstrap.cc:654] T 0ae4a92fdcdd41a787a3cdf620212d38 P 38ddfa85ddec470ca8a999e852c88dd9: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:24.287101 10680 tablet_bootstrap.cc:654] T 0ae4a92fdcdd41a787a3cdf620212d38 P 5a3086abd30a4331a257b50f39adf55e: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:24.287521 10678 tablet_bootstrap.cc:492] T 0ae4a92fdcdd41a787a3cdf620212d38 P 38ddfa85ddec470ca8a999e852c88dd9: No bootstrap required, opened a new log
I20260618 23:24:24.287573 10678 ts_tablet_manager.cc:1397] T 0ae4a92fdcdd41a787a3cdf620212d38 P 38ddfa85ddec470ca8a999e852c88dd9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:24.287676 10678 raft_consensus.cc:348] T 0ae4a92fdcdd41a787a3cdf620212d38 P 38ddfa85ddec470ca8a999e852c88dd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:24.287734 10680 tablet_bootstrap.cc:492] T 0ae4a92fdcdd41a787a3cdf620212d38 P 5a3086abd30a4331a257b50f39adf55e: No bootstrap required, opened a new log
I20260618 23:24:24.287750 10678 raft_consensus.cc:374] T 0ae4a92fdcdd41a787a3cdf620212d38 P 38ddfa85ddec470ca8a999e852c88dd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:24.287768 10680 ts_tablet_manager.cc:1397] T 0ae4a92fdcdd41a787a3cdf620212d38 P 5a3086abd30a4331a257b50f39adf55e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:24.287772 10678 raft_consensus.cc:729] T 0ae4a92fdcdd41a787a3cdf620212d38 P 38ddfa85ddec470ca8a999e852c88dd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38ddfa85ddec470ca8a999e852c88dd9, State: Initialized, Role: FOLLOWER
I20260618 23:24:24.287835 10678 consensus_queue.cc:260] T 0ae4a92fdcdd41a787a3cdf620212d38 P 38ddfa85ddec470ca8a999e852c88dd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:24.287917 10678 ts_tablet_manager.cc:1428] T 0ae4a92fdcdd41a787a3cdf620212d38 P 38ddfa85ddec470ca8a999e852c88dd9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260618 23:24:24.287910 10680 raft_consensus.cc:348] T 0ae4a92fdcdd41a787a3cdf620212d38 P 5a3086abd30a4331a257b50f39adf55e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:24.287976 10680 raft_consensus.cc:374] T 0ae4a92fdcdd41a787a3cdf620212d38 P 5a3086abd30a4331a257b50f39adf55e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:24.287992 10680 raft_consensus.cc:729] T 0ae4a92fdcdd41a787a3cdf620212d38 P 5a3086abd30a4331a257b50f39adf55e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a3086abd30a4331a257b50f39adf55e, State: Initialized, Role: FOLLOWER
I20260618 23:24:24.288064 10680 consensus_queue.cc:260] T 0ae4a92fdcdd41a787a3cdf620212d38 P 5a3086abd30a4331a257b50f39adf55e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:24.288148 10680 ts_tablet_manager.cc:1428] T 0ae4a92fdcdd41a787a3cdf620212d38 P 5a3086abd30a4331a257b50f39adf55e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:24.288516 10668 ts_tablet_manager.cc:1918] T bbca4a2238e34b26833a326d105922d9 P 38ddfa85ddec470ca8a999e852c88dd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 23:24:24.288574 10668 log.cc:1196] T bbca4a2238e34b26833a326d105922d9 P 38ddfa85ddec470ca8a999e852c88dd9: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-2/wal/wals/bbca4a2238e34b26833a326d105922d9
I20260618 23:24:24.288793 10668 ts_tablet_manager.cc:1939] T bbca4a2238e34b26833a326d105922d9 P 38ddfa85ddec470ca8a999e852c88dd9: Deleting consensus metadata
I20260618 23:24:24.289252 10067 catalog_manager.cc:4841] TS 38ddfa85ddec470ca8a999e852c88dd9 (127.8.39.131:38833): tablet bbca4a2238e34b26833a326d105922d9 (table test-table [id=9488d99086aa4e11abf008cf64d1990d]) successfully deleted
I20260618 23:24:24.289422 10667 ts_tablet_manager.cc:1905] T bbca4a2238e34b26833a326d105922d9 P 1118c3f91fbe492cadf521766345707a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:24.290753 10679 tablet_bootstrap.cc:492] T 0ae4a92fdcdd41a787a3cdf620212d38 P 1118c3f91fbe492cadf521766345707a: No bootstrap required, opened a new log
I20260618 23:24:24.290795 10679 ts_tablet_manager.cc:1397] T 0ae4a92fdcdd41a787a3cdf620212d38 P 1118c3f91fbe492cadf521766345707a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260618 23:24:24.290813 10667 ts_tablet_manager.cc:1918] T bbca4a2238e34b26833a326d105922d9 P 1118c3f91fbe492cadf521766345707a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 23:24:24.290875 10667 log.cc:1196] T bbca4a2238e34b26833a326d105922d9 P 1118c3f91fbe492cadf521766345707a: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-0/wal/wals/bbca4a2238e34b26833a326d105922d9
I20260618 23:24:24.290922 10679 raft_consensus.cc:348] T 0ae4a92fdcdd41a787a3cdf620212d38 P 1118c3f91fbe492cadf521766345707a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:24.290997 10679 raft_consensus.cc:374] T 0ae4a92fdcdd41a787a3cdf620212d38 P 1118c3f91fbe492cadf521766345707a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:24.291011 10679 raft_consensus.cc:729] T 0ae4a92fdcdd41a787a3cdf620212d38 P 1118c3f91fbe492cadf521766345707a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1118c3f91fbe492cadf521766345707a, State: Initialized, Role: FOLLOWER
I20260618 23:24:24.291083 10679 consensus_queue.cc:260] T 0ae4a92fdcdd41a787a3cdf620212d38 P 1118c3f91fbe492cadf521766345707a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:24.291134 10667 ts_tablet_manager.cc:1939] T bbca4a2238e34b26833a326d105922d9 P 1118c3f91fbe492cadf521766345707a: Deleting consensus metadata
I20260618 23:24:24.291177 10679 ts_tablet_manager.cc:1428] T 0ae4a92fdcdd41a787a3cdf620212d38 P 1118c3f91fbe492cadf521766345707a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:24.291420 10067 catalog_manager.cc:4841] TS 1118c3f91fbe492cadf521766345707a (127.8.39.129:41901): tablet bbca4a2238e34b26833a326d105922d9 (table test-table [id=9488d99086aa4e11abf008cf64d1990d]) successfully deleted
I20260618 23:24:24.370471 10684 raft_consensus.cc:482] T 0ae4a92fdcdd41a787a3cdf620212d38 P 5a3086abd30a4331a257b50f39adf55e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:24:24.370579 10684 raft_consensus.cc:504] T 0ae4a92fdcdd41a787a3cdf620212d38 P 5a3086abd30a4331a257b50f39adf55e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:24.370951 10684 leader_election.cc:288] T 0ae4a92fdcdd41a787a3cdf620212d38 P 5a3086abd30a4331a257b50f39adf55e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1118c3f91fbe492cadf521766345707a (127.8.39.129:41901), 38ddfa85ddec470ca8a999e852c88dd9 (127.8.39.131:38833)
I20260618 23:24:24.375329 10687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:24.371333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:60761 (local address 127.8.39.129:41901)
0618 23:24:24.371451 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:24.371453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:24.371466 (+ 13us) server_negotiation.cc:407] Connection header received
0618 23:24:24.371503 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:24.371504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:24.371517 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:24.371607 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:24.371781 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:24.372753 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:24.374336 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:24.374906 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:24.374992 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:24.375001 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:24.375220 (+ 219us) server_negotiation.cc:299] Negotiation successful
0618 23:24:24.375258 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260618 23:24:24.375345 10696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:24.371543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:35907 (local address 127.8.39.131:38833)
0618 23:24:24.371654 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:24.371656 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:24.371664 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:24:24.371704 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:24.371705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:24.371714 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:24.371821 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:24.372189 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:24.372893 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:24.374336 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:24.374920 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:24.374993 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:24.375001 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:24.375219 (+ 218us) server_negotiation.cc:299] Negotiation successful
0618 23:24:24.375260 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":54,"threads_started":1}
I20260618 23:24:24.375389 10695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:24.371411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:38833 (local address 127.8.39.130:35907)
0618 23:24:24.371538 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:24.371552 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:24.371611 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:24.371861 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:24.371862 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:24.371865 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:24.371931 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:24.371932 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:24.373034 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:24.373035 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:24.374221 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:24.374223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:24.375038 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:24.375039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:24.375125 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:24.375130 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:24:24.375200 (+ 70us) client_negotiation.cc:240] Negotiation successful
0618 23:24:24.375252 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":6,"thread_start_us":72,"threads_started":1}
I20260618 23:24:24.375391 10694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:24.371224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:41901 (local address 127.8.39.130:60761)
0618 23:24:24.371345 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:24.371358 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:24.371412 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:24.371620 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:24.371621 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:24.371624 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:24.371702 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:24.371704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:24.372865 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:24.372866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:24.374233 (+ 1367us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:24.374235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:24.375039 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:24.375040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:24.375125 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:24.375132 (+ 7us) client_negotiation.cc:769] Sending connection context
0618 23:24:24.375200 (+ 68us) client_negotiation.cc:240] Negotiation successful
0618 23:24:24.375252 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":37,"thread_start_us":75,"threads_started":1}
I20260618 23:24:24.375689 10501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ae4a92fdcdd41a787a3cdf620212d38" candidate_uuid: "5a3086abd30a4331a257b50f39adf55e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ddfa85ddec470ca8a999e852c88dd9" is_pre_election: true
I20260618 23:24:24.375777 10501 raft_consensus.cc:2454] T 0ae4a92fdcdd41a787a3cdf620212d38 P 38ddfa85ddec470ca8a999e852c88dd9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a3086abd30a4331a257b50f39adf55e in term 0.
I20260618 23:24:24.375746 10238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ae4a92fdcdd41a787a3cdf620212d38" candidate_uuid: "5a3086abd30a4331a257b50f39adf55e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1118c3f91fbe492cadf521766345707a" is_pre_election: true
I20260618 23:24:24.375859 10238 raft_consensus.cc:2454] T 0ae4a92fdcdd41a787a3cdf620212d38 P 1118c3f91fbe492cadf521766345707a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5a3086abd30a4331a257b50f39adf55e in term 0.
I20260618 23:24:24.375952 10311 leader_election.cc:302] T 0ae4a92fdcdd41a787a3cdf620212d38 P 5a3086abd30a4331a257b50f39adf55e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38ddfa85ddec470ca8a999e852c88dd9, 5a3086abd30a4331a257b50f39adf55e; no voters:
I20260618 23:24:24.376142 10684 raft_consensus.cc:2781] T 0ae4a92fdcdd41a787a3cdf620212d38 P 5a3086abd30a4331a257b50f39adf55e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:24:24.376191 10684 raft_consensus.cc:482] T 0ae4a92fdcdd41a787a3cdf620212d38 P 5a3086abd30a4331a257b50f39adf55e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:24:24.376209 10684 raft_consensus.cc:3037] T 0ae4a92fdcdd41a787a3cdf620212d38 P 5a3086abd30a4331a257b50f39adf55e [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:24.376901 10684 raft_consensus.cc:504] T 0ae4a92fdcdd41a787a3cdf620212d38 P 5a3086abd30a4331a257b50f39adf55e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:24.377009 10684 leader_election.cc:288] T 0ae4a92fdcdd41a787a3cdf620212d38 P 5a3086abd30a4331a257b50f39adf55e [CANDIDATE]: Term 1 election: Requested vote from peers 1118c3f91fbe492cadf521766345707a (127.8.39.129:41901), 38ddfa85ddec470ca8a999e852c88dd9 (127.8.39.131:38833)
I20260618 23:24:24.377214 10238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ae4a92fdcdd41a787a3cdf620212d38" candidate_uuid: "5a3086abd30a4331a257b50f39adf55e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1118c3f91fbe492cadf521766345707a"
I20260618 23:24:24.377240 10501 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ae4a92fdcdd41a787a3cdf620212d38" candidate_uuid: "5a3086abd30a4331a257b50f39adf55e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38ddfa85ddec470ca8a999e852c88dd9"
I20260618 23:24:24.377291 10238 raft_consensus.cc:3037] T 0ae4a92fdcdd41a787a3cdf620212d38 P 1118c3f91fbe492cadf521766345707a [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:24.377312 10501 raft_consensus.cc:3037] T 0ae4a92fdcdd41a787a3cdf620212d38 P 38ddfa85ddec470ca8a999e852c88dd9 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:24.377858 10501 raft_consensus.cc:2454] T 0ae4a92fdcdd41a787a3cdf620212d38 P 38ddfa85ddec470ca8a999e852c88dd9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a3086abd30a4331a257b50f39adf55e in term 1.
I20260618 23:24:24.377900 10238 raft_consensus.cc:2454] T 0ae4a92fdcdd41a787a3cdf620212d38 P 1118c3f91fbe492cadf521766345707a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5a3086abd30a4331a257b50f39adf55e in term 1.
I20260618 23:24:24.378046 10311 leader_election.cc:302] T 0ae4a92fdcdd41a787a3cdf620212d38 P 5a3086abd30a4331a257b50f39adf55e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1118c3f91fbe492cadf521766345707a, 5a3086abd30a4331a257b50f39adf55e; no voters:
I20260618 23:24:24.378154 10684 raft_consensus.cc:2781] T 0ae4a92fdcdd41a787a3cdf620212d38 P 5a3086abd30a4331a257b50f39adf55e [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:24.378314 10684 raft_consensus.cc:686] T 0ae4a92fdcdd41a787a3cdf620212d38 P 5a3086abd30a4331a257b50f39adf55e [term 1 LEADER]: Becoming Leader. State: Replica: 5a3086abd30a4331a257b50f39adf55e, State: Running, Role: LEADER
I20260618 23:24:24.378396 10684 consensus_queue.cc:237] T 0ae4a92fdcdd41a787a3cdf620212d38 P 5a3086abd30a4331a257b50f39adf55e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } }
I20260618 23:24:24.379319 10079 catalog_manager.cc:5495] T 0ae4a92fdcdd41a787a3cdf620212d38 P 5a3086abd30a4331a257b50f39adf55e reported cstate change: term changed from 0 to 1, leader changed from <none> to 5a3086abd30a4331a257b50f39adf55e (127.8.39.130). New cstate: current_term: 1 leader_uuid: "5a3086abd30a4331a257b50f39adf55e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a3086abd30a4331a257b50f39adf55e" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37285 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 } health_report { overall_health: UNKNOWN } } }
I20260618 23:24:24.410826 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:24.407510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36914 (local address 127.8.39.130:37285)
0618 23:24:24.407684 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:24.407686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:24.407695 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:24:24.407752 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:24.407753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:24.407772 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:24.407854 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:24:24.408426 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:24.408940 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:24.409761 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:24.410027 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:24.410232 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:24.410456 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:24:24.410566 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:24:24.410635 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:24.410728 (+ 93us) server_negotiation.cc:299] Negotiation successful
0618 23:24:24.410751 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":114,"threads_started":1}
I20260618 23:24:24.415721 10696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:24.413500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:34849 (local address 127.8.39.131:38833)
0618 23:24:24.413538 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:24.413540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:24.413586 (+ 46us) server_negotiation.cc:407] Connection header received
0618 23:24:24.413634 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:24.413635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:24.413644 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:24.413689 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:24:24.413802 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:24.414265 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:24.415033 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:24.415264 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:24.415323 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:24.415463 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:24:24.415487 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:24:24.415531 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:24.415640 (+ 109us) server_negotiation.cc:299] Negotiation successful
0618 23:24:24.415660 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260618 23:24:24.415725 10702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:24.413370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:38833 (local address 127.8.39.130:34849)
0618 23:24:24.413528 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:24.413539 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:24.413596 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:24.413701 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:24.413701 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:24.413703 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 23:24:24.413752 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:24.413753 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:24.414337 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:24.414337 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:24.414891 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:24.414893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:24.415337 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:24.415337 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:24.415383 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:24.415394 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 23:24:24.415541 (+ 147us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 23:24:24.415542 (+ 1us) client_negotiation.cc:769] Sending connection context
0618 23:24:24.415624 (+ 82us) client_negotiation.cc:240] Negotiation successful
0618 23:24:24.415666 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
I20260618 23:24:24.417855 10501 raft_consensus.cc:1261] T 0ae4a92fdcdd41a787a3cdf620212d38 P 38ddfa85ddec470ca8a999e852c88dd9 [term 1 FOLLOWER]: Refusing update from remote peer 5a3086abd30a4331a257b50f39adf55e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:24:24.417925 10238 raft_consensus.cc:1261] T 0ae4a92fdcdd41a787a3cdf620212d38 P 1118c3f91fbe492cadf521766345707a [term 1 FOLLOWER]: Refusing update from remote peer 5a3086abd30a4331a257b50f39adf55e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:24:24.418105 10697 consensus_queue.cc:1035] T 0ae4a92fdcdd41a787a3cdf620212d38 P 5a3086abd30a4331a257b50f39adf55e [LEADER]: Connected to new peer: Peer: permanent_uuid: "38ddfa85ddec470ca8a999e852c88dd9" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 38833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:24.418247 10684 consensus_queue.cc:1035] T 0ae4a92fdcdd41a787a3cdf620212d38 P 5a3086abd30a4331a257b50f39adf55e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1118c3f91fbe492cadf521766345707a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41901 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:24.424644 10699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:24.422416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:56577 (local address 127.8.39.130:37285)
0618 23:24:24.422455 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:24.422457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:24.422495 (+ 38us) server_negotiation.cc:407] Connection header received
0618 23:24:24.422570 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:24.422571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:24.422581 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:24.422622 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:24:24.422752 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:24.423218 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:24.423959 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:24.424278 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:24.424349 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:24.424486 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:24:24.424510 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:24:24.424550 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:24.424580 (+ 30us) server_negotiation.cc:299] Negotiation successful
0618 23:24:24.424595 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260618 23:24:24.424662 10707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:24.422318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:37285 (local address 127.8.39.131:56577)
0618 23:24:24.422442 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:24.422453 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:24.422509 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:24.422636 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:24.422637 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:24.422639 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 23:24:24.422683 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:24.422684 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:24.423302 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:24.423303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:24.423860 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:24.423861 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:24.424349 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:24.424350 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:24.424425 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:24.424430 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 23:24:24.424547 (+ 117us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 23:24:24.424548 (+ 1us) client_negotiation.cc:769] Sending connection context
0618 23:24:24.424578 (+ 30us) client_negotiation.cc:240] Negotiation successful
0618 23:24:24.424611 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":77,"threads_started":1}
W20260618 23:24:24.424990 10649 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260618 23:24:25.948758 10079 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:24:25.948897 10079 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:24:25.950204 10079 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a7b49d74648e4730ba179de0645341e5: Sending DeleteTablet for 3 replicas of tablet 0ae4a92fdcdd41a787a3cdf620212d38
I20260618 23:24:25.950495 10218 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ae4a92fdcdd41a787a3cdf620212d38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:25 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:56268
I20260618 23:24:25.950577 10358 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ae4a92fdcdd41a787a3cdf620212d38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:25 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:35636
I20260618 23:24:25.950700 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 10153
I20260618 23:24:25.950749 10498 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ae4a92fdcdd41a787a3cdf620212d38 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:25 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:48354
I20260618 23:24:25.951694 10732 tablet_replica.cc:330] T 0ae4a92fdcdd41a787a3cdf620212d38 P 38ddfa85ddec470ca8a999e852c88dd9: stopping tablet replica
I20260618 23:24:25.951756 10732 raft_consensus.cc:2229] T 0ae4a92fdcdd41a787a3cdf620212d38 P 38ddfa85ddec470ca8a999e852c88dd9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:24:25.951795 10732 raft_consensus.cc:2258] T 0ae4a92fdcdd41a787a3cdf620212d38 P 38ddfa85ddec470ca8a999e852c88dd9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:25.952153 10732 ts_tablet_manager.cc:1905] T 0ae4a92fdcdd41a787a3cdf620212d38 P 38ddfa85ddec470ca8a999e852c88dd9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:25.953964 10732 ts_tablet_manager.cc:1918] T 0ae4a92fdcdd41a787a3cdf620212d38 P 38ddfa85ddec470ca8a999e852c88dd9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260618 23:24:25.954018 10732 log.cc:1196] T 0ae4a92fdcdd41a787a3cdf620212d38 P 38ddfa85ddec470ca8a999e852c88dd9: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781825053542692-8350-0/minicluster-data/ts-2/wal/wals/0ae4a92fdcdd41a787a3cdf620212d38
I20260618 23:24:25.954237 10732 ts_tablet_manager.cc:1939] T 0ae4a92fdcdd41a787a3cdf620212d38 P 38ddfa85ddec470ca8a999e852c88dd9: Deleting consensus metadata
I20260618 23:24:25.954613 10067 catalog_manager.cc:4841] TS 38ddfa85ddec470ca8a999e852c88dd9 (127.8.39.131:38833): tablet 0ae4a92fdcdd41a787a3cdf620212d38 (table test-table [id=5033b0ebf89e4b4bb351a7f963d06441]) successfully deleted
I20260618 23:24:25.956626 10731 tablet_replica.cc:330] T 0ae4a92fdcdd41a787a3cdf620212d38 P 5a3086abd30a4331a257b50f39adf55e: stopping tablet replica
I20260618 23:24:25.956699 10731 raft_consensus.cc:2229] T 0ae4a92fdcdd41a787a3cdf620212d38 P 5a3086abd30a4331a257b50f39adf55e [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:24:25.956800 10731 raft_consensus.cc:2258] T 0ae4a92fdcdd41a787a3cdf620212d38 P 5a3086abd30a4331a257b50f39adf55e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:25.957033 10731 ts_tablet_manager.cc:1905] T 0ae4a92fdcdd41a787a3cdf620212d38 P 5a3086abd30a4331a257b50f39adf55e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260618 23:24:25.958011 10067 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.39.129:41901 (error 108)
I20260618 23:24:25.958259 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 10293
I20260618 23:24:25.958968 10734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:25.958392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:41901 (local address 127.0.0.1:55102)
0618 23:24:25.958884 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:25.958929 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.129:41901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":58,"threads_started":1}
W20260618 23:24:25.959105 10067 catalog_manager.cc:4584] TS 1118c3f91fbe492cadf521766345707a (127.8.39.129:41901): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0ae4a92fdcdd41a787a3cdf620212d38: Network error: Client connection negotiation failed: client connection to 127.8.39.129:41901: connect: Connection refused (error 111)
I20260618 23:24:25.964974 10734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:25.964858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:37285 (local address 127.0.0.1:36920)
0618 23:24:25.964913 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:25.964939 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.130:37285: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260618 23:24:25.964964 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 10433
I20260618 23:24:25.971133 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 10049
2026-06-18T23:24:25Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5946 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:24:26 dist-test-slave-47qt krb5kdc[10737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:24:26 dist-test-slave-47qt krb5kdc[10737](info): set up 2 sockets
Jun 18 23:24:26 dist-test-slave-47qt krb5kdc[10737](info): commencing operation
krb5kdc: starting...
W20260618 23:24:27.033958 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:24:27 dist-test-slave-47qt krb5kdc[10737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:24:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:24:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.39.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
I20260618 23:24:27.156770 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:46169
--webserver_interface=127.8.39.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:38761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:46169
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:27.236905 10750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:27.237138 10750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:27.237203 10750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:27.238339 10750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:24:27.238413 10750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:27.238440 10750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:27.238466 10750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:24:27.238485 10750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:24:27.239913 10750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:38761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:46169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:46169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:27.240186 10750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:27.240403 10750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:27.246042 10758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:27.246178 10755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:27.246160 10756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:27.246209 10750 server_base.cc:1034] running on GCE node
I20260618 23:24:27.246497 10750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:27.246780 10750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:27.247957 10750 hybrid_clock.cc:648] HybridClock initialized: now 1781825067247940 us; error 49 us; skew 500 ppm
Jun 18 23:24:27 dist-test-slave-47qt krb5kdc[10737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825067, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:27.250412 10750 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:24:27.250975 10750 webserver.cc:460] Webserver started at http://127.8.39.190:33487/ using document root <none> and password file <none>
I20260618 23:24:27.251232 10750 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:27.251302 10750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:27.251446 10750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:27.252473 10750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "c747a40d516c4d77a2dd3c605bbe1d39"
format_stamp: "Formatted at 2026-06-18 23:24:27 on dist-test-slave-47qt"
server_key: "4f51374cc0d521e0ec9852fff99f10a9"
server_key_iv: "4bf19d4b7de6b8516ac11168551b64f0"
server_key_version: "encryptionkey@0"
I20260618 23:24:27.252830 10750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "c747a40d516c4d77a2dd3c605bbe1d39"
format_stamp: "Formatted at 2026-06-18 23:24:27 on dist-test-slave-47qt"
server_key: "4f51374cc0d521e0ec9852fff99f10a9"
server_key_iv: "4bf19d4b7de6b8516ac11168551b64f0"
server_key_version: "encryptionkey@0"
I20260618 23:24:27.254419 10750 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:27.255210 10765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:27.255391 10750 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:27.255491 10750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "c747a40d516c4d77a2dd3c605bbe1d39"
format_stamp: "Formatted at 2026-06-18 23:24:27 on dist-test-slave-47qt"
server_key: "4f51374cc0d521e0ec9852fff99f10a9"
server_key_iv: "4bf19d4b7de6b8516ac11168551b64f0"
server_key_version: "encryptionkey@0"
I20260618 23:24:27.255579 10750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:27.299007 10750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:27.301577 10750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:27.301713 10750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:27.306692 10750 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:46169
I20260618 23:24:27.306811 10817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:46169 every 8 connection(s)
I20260618 23:24:27.307312 10750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:24:27.308238 10818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:27.310628 10818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c747a40d516c4d77a2dd3c605bbe1d39: Bootstrap starting.
I20260618 23:24:27.311201 10818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c747a40d516c4d77a2dd3c605bbe1d39: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:27.311489 10818 log.cc:824] T 00000000000000000000000000000000 P c747a40d516c4d77a2dd3c605bbe1d39: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:27.311848 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 10750
I20260618 23:24:27.311929 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:24:27.312094 8350 external_mini_cluster.cc:1428] Setting key 657b1d66eaff0bcac6b278d5d3b53a83
I20260618 23:24:27.312631 10818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c747a40d516c4d77a2dd3c605bbe1d39: No bootstrap required, opened a new log
I20260618 23:24:27.314881 10818 raft_consensus.cc:348] T 00000000000000000000000000000000 P c747a40d516c4d77a2dd3c605bbe1d39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c747a40d516c4d77a2dd3c605bbe1d39" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 46169 } }
I20260618 23:24:27.315038 10818 raft_consensus.cc:374] T 00000000000000000000000000000000 P c747a40d516c4d77a2dd3c605bbe1d39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:27.315084 10818 raft_consensus.cc:729] T 00000000000000000000000000000000 P c747a40d516c4d77a2dd3c605bbe1d39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c747a40d516c4d77a2dd3c605bbe1d39, State: Initialized, Role: FOLLOWER
I20260618 23:24:27.315228 10818 consensus_queue.cc:260] T 00000000000000000000000000000000 P c747a40d516c4d77a2dd3c605bbe1d39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c747a40d516c4d77a2dd3c605bbe1d39" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 46169 } }
I20260618 23:24:27.315333 10818 raft_consensus.cc:388] T 00000000000000000000000000000000 P c747a40d516c4d77a2dd3c605bbe1d39 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:24:27.315378 10818 raft_consensus.cc:482] T 00000000000000000000000000000000 P c747a40d516c4d77a2dd3c605bbe1d39 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:24:27.315425 10818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c747a40d516c4d77a2dd3c605bbe1d39 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:27.315991 10818 raft_consensus.cc:504] T 00000000000000000000000000000000 P c747a40d516c4d77a2dd3c605bbe1d39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c747a40d516c4d77a2dd3c605bbe1d39" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 46169 } }
I20260618 23:24:27.316125 10818 leader_election.cc:302] T 00000000000000000000000000000000 P c747a40d516c4d77a2dd3c605bbe1d39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c747a40d516c4d77a2dd3c605bbe1d39; no voters:
I20260618 23:24:27.316316 10818 leader_election.cc:288] T 00000000000000000000000000000000 P c747a40d516c4d77a2dd3c605bbe1d39 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:24:27.316352 10823 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c747a40d516c4d77a2dd3c605bbe1d39 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:27.316520 10823 raft_consensus.cc:686] T 00000000000000000000000000000000 P c747a40d516c4d77a2dd3c605bbe1d39 [term 1 LEADER]: Becoming Leader. State: Replica: c747a40d516c4d77a2dd3c605bbe1d39, State: Running, Role: LEADER
I20260618 23:24:27.316572 10818 sys_catalog.cc:564] T 00000000000000000000000000000000 P c747a40d516c4d77a2dd3c605bbe1d39 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:24:27.316689 10823 consensus_queue.cc:237] T 00000000000000000000000000000000 P c747a40d516c4d77a2dd3c605bbe1d39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c747a40d516c4d77a2dd3c605bbe1d39" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 46169 } }
I20260618 23:24:27.317157 10824 sys_catalog.cc:455] T 00000000000000000000000000000000 P c747a40d516c4d77a2dd3c605bbe1d39 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c747a40d516c4d77a2dd3c605bbe1d39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c747a40d516c4d77a2dd3c605bbe1d39" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 46169 } } }
I20260618 23:24:27.317243 10824 sys_catalog.cc:458] T 00000000000000000000000000000000 P c747a40d516c4d77a2dd3c605bbe1d39 [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:27.317454 10825 sys_catalog.cc:455] T 00000000000000000000000000000000 P c747a40d516c4d77a2dd3c605bbe1d39 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c747a40d516c4d77a2dd3c605bbe1d39. Latest consensus state: current_term: 1 leader_uuid: "c747a40d516c4d77a2dd3c605bbe1d39" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c747a40d516c4d77a2dd3c605bbe1d39" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 46169 } } }
I20260618 23:24:27.317727 10825 sys_catalog.cc:458] T 00000000000000000000000000000000 P c747a40d516c4d77a2dd3c605bbe1d39 [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:27.317950 10830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:24:27.318601 10830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:24:27.320255 10830 catalog_manager.cc:1347] Generated new cluster ID: c23ff2337eea4cfbab234177003cbf1e
Jun 18 23:24:27 dist-test-slave-47qt krb5kdc[10737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:27.320307 10830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:24:27.324649 10822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:27.312603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59640 (local address 127.8.39.190:46169)
0618 23:24:27.312807 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:27.312814 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:27.312837 (+ 23us) server_negotiation.cc:407] Connection header received
0618 23:24:27.313851 (+ 1014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:27.313858 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:27.313903 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:27.314010 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:27.314802 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:27.316148 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:27.317270 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:27.317909 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:27.318007 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:27.320772 (+ 2765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:27.320775 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:27.320775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:27.320787 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:27.322782 (+ 1995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:27.323163 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:27.323166 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:27.323166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:27.323270 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:27.323470 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:27.323471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:27.323471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:27.323891 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:27.324060 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:27.324135 (+ 75us) server_negotiation.cc:299] Negotiation successful
0618 23:24:27.324283 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":98,"threads_started":1}
I20260618 23:24:27.325640 10830 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:24:27.326308 10830 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:24:27.335304 10830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c747a40d516c4d77a2dd3c605bbe1d39: Generated new TSK 0
I20260618 23:24:27.335552 10830 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.39.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
I20260618 23:24:27.364480 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.129:0
--local_ip_for_outbound_sockets=127.8.39.129
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:46169
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:38761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:27.447656 10844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:27.447862 10844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:27.447886 10844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:27.449229 10844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:27.449272 10844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:27.449301 10844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.129
I20260618 23:24:27.450727 10844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:38761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:46169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10844
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.129
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:27.450963 10844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:27.451205 10844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:27.456032 10850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:27.456053 10852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:27.456285 10849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:27.456259 10844 server_base.cc:1034] running on GCE node
I20260618 23:24:27.456501 10844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:27.456687 10844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:27.457856 10844 hybrid_clock.cc:648] HybridClock initialized: now 1781825067457839 us; error 28 us; skew 500 ppm
Jun 18 23:24:27 dist-test-slave-47qt krb5kdc[10737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825067, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:27.460418 10844 init.cc:377] Logged in from keytab as kudu/127.8.39.129@KRBTEST.COM (short username kudu)
I20260618 23:24:27.460819 10844 webserver.cc:460] Webserver started at http://127.8.39.129:39765/ using document root <none> and password file <none>
I20260618 23:24:27.461040 10844 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:27.461107 10844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:27.461287 10844 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:27.462301 10844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/data/instance:
uuid: "c5971d0cb80940669349c7221ed6c6c5"
format_stamp: "Formatted at 2026-06-18 23:24:27 on dist-test-slave-47qt"
server_key: "a81741213cfe2cee0523932d82d77a3b"
server_key_iv: "71695b2c534a9e8d5172604d0521403d"
server_key_version: "encryptionkey@0"
I20260618 23:24:27.462652 10844 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance:
uuid: "c5971d0cb80940669349c7221ed6c6c5"
format_stamp: "Formatted at 2026-06-18 23:24:27 on dist-test-slave-47qt"
server_key: "a81741213cfe2cee0523932d82d77a3b"
server_key_iv: "71695b2c534a9e8d5172604d0521403d"
server_key_version: "encryptionkey@0"
I20260618 23:24:27.464489 10844 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260618 23:24:27.465366 10859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:27.465575 10844 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:27.465677 10844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/wal
uuid: "c5971d0cb80940669349c7221ed6c6c5"
format_stamp: "Formatted at 2026-06-18 23:24:27 on dist-test-slave-47qt"
server_key: "a81741213cfe2cee0523932d82d77a3b"
server_key_iv: "71695b2c534a9e8d5172604d0521403d"
server_key_version: "encryptionkey@0"
I20260618 23:24:27.465755 10844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:27.483835 10844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:27.486388 10844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:27.486568 10844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:27.486795 10844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:27.487102 10844 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:27.487159 10844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:27.487200 10844 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:27.487226 10844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:27.493022 10844 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.129:43753
I20260618 23:24:27.493063 10972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.129:43753 every 8 connection(s)
I20260618 23:24:27.493402 10844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
I20260618 23:24:27.499418 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 10844
I20260618 23:24:27.499491 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance
I20260618 23:24:27.499713 8350 external_mini_cluster.cc:1428] Setting key 823d6b0b16d406c42f09b907a8fd5011
Jun 18 23:24:27 dist-test-slave-47qt krb5kdc[10737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825067, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:27.504034 10975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:27.494271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:46169 (local address 127.8.39.129:44577)
0618 23:24:27.494449 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:27.494482 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:27.494981 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:27.495217 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:27.495232 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:27.495774 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:27.495954 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:27.495958 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:27.496653 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:27.496654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:27.497458 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:27.497460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:27.497877 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:27.497878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:27.497993 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:27.498457 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:27.498466 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:27.500221 (+ 1755us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:27.502170 (+ 1949us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:27.502177 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:27.502179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:27.502494 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:27.502809 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:27.502810 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:27.502810 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:27.502954 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:27.503292 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:27.503298 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:27.503463 (+ 165us) client_negotiation.cc:769] Sending connection context
0618 23:24:27.503580 (+ 117us) client_negotiation.cc:240] Negotiation successful
0618 23:24:27.503755 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":104,"threads_started":1}
I20260618 23:24:27.504138 10822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:27.494274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:44577 (local address 127.8.39.190:46169)
0618 23:24:27.494324 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:27.494326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:27.494571 (+ 245us) server_negotiation.cc:407] Connection header received
0618 23:24:27.495085 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:27.495087 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:27.495098 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:27.495147 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:27.496072 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:27.496562 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:27.497552 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:27.497778 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:27.497867 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:27.500347 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:27.500349 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:27.500349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:27.500357 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:27.502072 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:27.502664 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:27.502665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:27.502666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:27.502727 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:27.503071 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:27.503073 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:27.503073 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:27.503202 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:27.503280 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:27.503705 (+ 425us) server_negotiation.cc:299] Negotiation successful
0618 23:24:27.503996 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260618 23:24:27.504803 10973 heartbeater.cc:347] Connected to a master server at 127.8.39.190:46169
I20260618 23:24:27.504984 10973 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:27.505302 10973 heartbeater.cc:511] Master 127.8.39.190:46169 requested a full tablet report, sending...
I20260618 23:24:27.506006 10782 ts_manager.cc:194] Registered new tserver with Master: c5971d0cb80940669349c7221ed6c6c5 (127.8.39.129:43753)
I20260618 23:24:27.506891 10782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.129@KRBTEST.COM'} at 127.8.39.129:44577
WARNING: no policy specified for kudu/127.8.39.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
I20260618 23:24:27.524488 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.130:0
--local_ip_for_outbound_sockets=127.8.39.130
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:46169
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:38761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:27.603885 10978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:27.604128 10978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:27.604168 10978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:27.605844 10978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:27.605919 10978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:27.605966 10978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.130
I20260618 23:24:27.608095 10978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:38761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:46169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.130
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:27.608385 10978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:27.608608 10978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:27.613942 10986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:27.614028 10984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:27.614078 10978 server_base.cc:1034] running on GCE node
W20260618 23:24:27.614169 10983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:27.614387 10978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:27.614629 10978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:27.615783 10978 hybrid_clock.cc:648] HybridClock initialized: now 1781825067615762 us; error 34 us; skew 500 ppm
Jun 18 23:24:27 dist-test-slave-47qt krb5kdc[10737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825067, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:27.618431 10978 init.cc:377] Logged in from keytab as kudu/127.8.39.130@KRBTEST.COM (short username kudu)
I20260618 23:24:27.618911 10978 webserver.cc:460] Webserver started at http://127.8.39.130:42537/ using document root <none> and password file <none>
I20260618 23:24:27.619165 10978 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:27.619236 10978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:27.619371 10978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:27.620385 10978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/data/instance:
uuid: "9ff7b7b7a7af4508b5090111b7f7db23"
format_stamp: "Formatted at 2026-06-18 23:24:27 on dist-test-slave-47qt"
server_key: "f0a6efc6a3c63484f5c634269c10bf39"
server_key_iv: "7eb9437929177346010d58fc3d31ed58"
server_key_version: "encryptionkey@0"
I20260618 23:24:27.620754 10978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance:
uuid: "9ff7b7b7a7af4508b5090111b7f7db23"
format_stamp: "Formatted at 2026-06-18 23:24:27 on dist-test-slave-47qt"
server_key: "f0a6efc6a3c63484f5c634269c10bf39"
server_key_iv: "7eb9437929177346010d58fc3d31ed58"
server_key_version: "encryptionkey@0"
I20260618 23:24:27.622125 10978 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:24:27.622884 10993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:27.623057 10978 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260618 23:24:27.623154 10978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/wal
uuid: "9ff7b7b7a7af4508b5090111b7f7db23"
format_stamp: "Formatted at 2026-06-18 23:24:27 on dist-test-slave-47qt"
server_key: "f0a6efc6a3c63484f5c634269c10bf39"
server_key_iv: "7eb9437929177346010d58fc3d31ed58"
server_key_version: "encryptionkey@0"
I20260618 23:24:27.623245 10978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:27.633775 10978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:27.636013 10978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:27.636175 10978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:27.636392 10978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:27.636683 10978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:27.636739 10978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:27.636778 10978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:27.636806 10978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:27.642760 10978 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.130:37741
I20260618 23:24:27.642942 11106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.130:37741 every 8 connection(s)
I20260618 23:24:27.643178 10978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
I20260618 23:24:27.648840 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 10978
I20260618 23:24:27.648967 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance
I20260618 23:24:27.649197 8350 external_mini_cluster.cc:1428] Setting key da8cc5ec89ec1eaedfec1e0cb63a9513
Jun 18 23:24:27 dist-test-slave-47qt krb5kdc[10737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825067, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:27.652724 11109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:27.643964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:46169 (local address 127.8.39.130:45339)
0618 23:24:27.644160 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:27.644194 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:27.644771 (+ 577us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:27.645002 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:27.645005 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:27.645748 (+ 743us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:27.645936 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:27.645940 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:27.646589 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:27.646590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:27.647350 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:27.647352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:27.647673 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:27.647673 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:27.647764 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:27.648174 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:27.648181 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:27.649631 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:27.651495 (+ 1864us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:27.651500 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:27.651502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:27.651707 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:27.651911 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:27.651911 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:27.651912 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:27.652000 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:27.652262 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:27.652266 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:27.652370 (+ 104us) client_negotiation.cc:769] Sending connection context
0618 23:24:27.652449 (+ 79us) client_negotiation.cc:240] Negotiation successful
0618 23:24:27.652559 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":91,"threads_started":1}
I20260618 23:24:27.652845 10822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:27.643967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:45339 (local address 127.8.39.190:46169)
0618 23:24:27.644021 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:27.644023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:27.644278 (+ 255us) server_negotiation.cc:407] Connection header received
0618 23:24:27.644871 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:27.644872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:27.644881 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:27.644950 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:27.646026 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:27.646503 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:27.647424 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:27.647601 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:27.647654 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:27.649737 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:27.649738 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:27.649739 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:27.649746 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:27.651407 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:27.651782 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:27.651783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:27.651783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:27.651845 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:27.652048 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:27.652049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:27.652050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:27.652198 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:27.652268 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:27.652630 (+ 362us) server_negotiation.cc:299] Negotiation successful
0618 23:24:27.652721 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260618 23:24:27.653365 11107 heartbeater.cc:347] Connected to a master server at 127.8.39.190:46169
I20260618 23:24:27.653475 11107 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:27.653786 11107 heartbeater.cc:511] Master 127.8.39.190:46169 requested a full tablet report, sending...
I20260618 23:24:27.654230 10782 ts_manager.cc:194] Registered new tserver with Master: 9ff7b7b7a7af4508b5090111b7f7db23 (127.8.39.130:37741)
I20260618 23:24:27.654598 10782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.130@KRBTEST.COM'} at 127.8.39.130:45339
WARNING: no policy specified for kudu/127.8.39.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
I20260618 23:24:27.676676 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.131:0
--local_ip_for_outbound_sockets=127.8.39.131
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:46169
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:38761
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:27.759092 11112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:27.759289 11112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:27.759311 11112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:27.760437 11112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:27.760478 11112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:27.760504 11112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.131
I20260618 23:24:27.762095 11112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:38761
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:46169
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11112
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.131
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:27.762341 11112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:27.762599 11112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 23:24:27.768317 11112 server_base.cc:1034] running on GCE node
W20260618 23:24:27.768322 11117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:27.768309 11118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:27.768450 11120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:27.768808 11112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:27.769068 11112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:27.770285 11112 hybrid_clock.cc:648] HybridClock initialized: now 1781825067770272 us; error 28 us; skew 500 ppm
Jun 18 23:24:27 dist-test-slave-47qt krb5kdc[10737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825067, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:27.772771 11112 init.cc:377] Logged in from keytab as kudu/127.8.39.131@KRBTEST.COM (short username kudu)
I20260618 23:24:27.773283 11112 webserver.cc:460] Webserver started at http://127.8.39.131:35163/ using document root <none> and password file <none>
I20260618 23:24:27.773517 11112 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:27.773563 11112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:27.773706 11112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:27.774718 11112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/data/instance:
uuid: "1e83cc6c7ea64ea18425d7b50847e631"
format_stamp: "Formatted at 2026-06-18 23:24:27 on dist-test-slave-47qt"
server_key: "3f6fb3acaa38968c5516c466501ee15b"
server_key_iv: "1e13d5bbfdb01d80f7b491a717a2fe61"
server_key_version: "encryptionkey@0"
I20260618 23:24:27.775067 11112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance:
uuid: "1e83cc6c7ea64ea18425d7b50847e631"
format_stamp: "Formatted at 2026-06-18 23:24:27 on dist-test-slave-47qt"
server_key: "3f6fb3acaa38968c5516c466501ee15b"
server_key_iv: "1e13d5bbfdb01d80f7b491a717a2fe61"
server_key_version: "encryptionkey@0"
I20260618 23:24:27.776346 11112 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:24:27.777058 11127 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:27.777248 11112 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260618 23:24:27.777341 11112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/wal
uuid: "1e83cc6c7ea64ea18425d7b50847e631"
format_stamp: "Formatted at 2026-06-18 23:24:27 on dist-test-slave-47qt"
server_key: "3f6fb3acaa38968c5516c466501ee15b"
server_key_iv: "1e13d5bbfdb01d80f7b491a717a2fe61"
server_key_version: "encryptionkey@0"
I20260618 23:24:27.777433 11112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:27.804651 11112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:27.807093 11112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:27.807265 11112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:27.807509 11112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:27.807854 11112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:27.807919 11112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:27.807957 11112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:27.807983 11112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:27.814219 11112 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.131:43659
I20260618 23:24:27.814260 11240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.131:43659 every 8 connection(s)
I20260618 23:24:27.814569 11112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
Jun 18 23:24:27 dist-test-slave-47qt krb5kdc[10737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825067, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:27.821619 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 11112
I20260618 23:24:27.821728 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance
I20260618 23:24:27.821944 8350 external_mini_cluster.cc:1428] Setting key 154599868012bca67f3cee4c7a34cb71
I20260618 23:24:27.823820 10822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:27.815214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:39837 (local address 127.8.39.190:46169)
0618 23:24:27.815284 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:27.815286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:27.815501 (+ 215us) server_negotiation.cc:407] Connection header received
0618 23:24:27.816017 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:27.816018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:27.816030 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:27.816086 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:27.817045 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:27.817668 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:27.818706 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:27.818911 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:27.819005 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:27.821028 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:27.821029 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:27.821030 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:27.821036 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:27.822617 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:27.823070 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:27.823071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:27.823071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:27.823121 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:27.823281 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:27.823281 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:27.823281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:27.823370 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:27.823414 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:27.823697 (+ 283us) server_negotiation.cc:299] Negotiation successful
0618 23:24:27.823760 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260618 23:24:27.823891 11243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:27.815227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:46169 (local address 127.8.39.131:39837)
0618 23:24:27.815385 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:27.815417 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:27.815902 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:27.816156 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:27.816159 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:27.816761 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:27.816946 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:27.816950 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:27.817784 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:27.817785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:27.818601 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:27.818604 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:27.819015 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:27.819016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:27.819139 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:27.819654 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:27.819661 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:27.820926 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:27.822750 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:27.822757 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:27.822759 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:27.823012 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:27.823164 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:27.823164 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:27.823164 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:27.823246 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:27.823424 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:27.823428 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:27.823545 (+ 117us) client_negotiation.cc:769] Sending connection context
0618 23:24:27.823629 (+ 84us) client_negotiation.cc:240] Negotiation successful
0618 23:24:27.823742 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":101,"threads_started":1}
I20260618 23:24:27.824491 11241 heartbeater.cc:347] Connected to a master server at 127.8.39.190:46169
I20260618 23:24:27.824595 11241 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:27.824767 11241 heartbeater.cc:511] Master 127.8.39.190:46169 requested a full tablet report, sending...
I20260618 23:24:27.825155 10782 ts_manager.cc:194] Registered new tserver with Master: 1e83cc6c7ea64ea18425d7b50847e631 (127.8.39.131:43659)
I20260618 23:24:27.825505 10782 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.131@KRBTEST.COM'} at 127.8.39.131:39837
I20260618 23:24:27.833848 8350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260618 23:24:27.845196 10822 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:27.844279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59656 (local address 127.8.39.190:46169)
0618 23:24:27.844359 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:27.844362 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:27.844438 (+ 76us) server_negotiation.cc:407] Connection header received
0618 23:24:27.844491 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:27.844491 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:27.844500 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:27.844564 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:27.845075 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59656: BlockingRecv error: recv got EOF from 127.0.0.1:59656 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
I20260618 23:24:27.845976 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 10844
I20260618 23:24:27.850028 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 10978
I20260618 23:24:27.854998 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 11112
I20260618 23:24:27.859249 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 10750
2026-06-18T23:24:27Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1883 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:24:27 dist-test-slave-47qt krb5kdc[11253](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:24:27 dist-test-slave-47qt krb5kdc[11253](info): set up 2 sockets
Jun 18 23:24:27 dist-test-slave-47qt krb5kdc[11253](info): commencing operation
krb5kdc: starting...
W20260618 23:24:28.926045 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:24:29 dist-test-slave-47qt krb5kdc[11253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:24:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:24:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.39.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
I20260618 23:24:29.047912 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:42315
--webserver_interface=127.8.39.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:45237
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:42315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:29.129880 11266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:29.130137 11266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:29.130173 11266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:29.131304 11266 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:24:29.131373 11266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:29.131398 11266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:29.131426 11266 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:24:29.131455 11266 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:24:29.132910 11266 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:45237
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:42315
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:42315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:29.133213 11266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:29.133453 11266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:29.138398 11272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:29.138417 11274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:29.138744 11266 server_base.cc:1034] running on GCE node
W20260618 23:24:29.138443 11271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:29.139058 11266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:29.139338 11266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:29.140508 11266 hybrid_clock.cc:648] HybridClock initialized: now 1781825069140461 us; error 65 us; skew 500 ppm
Jun 18 23:24:29 dist-test-slave-47qt krb5kdc[11253](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825069, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:29.143153 11266 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:24:29.143615 11266 webserver.cc:460] Webserver started at http://127.8.39.190:37115/ using document root <none> and password file <none>
I20260618 23:24:29.143811 11266 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:29.143855 11266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:29.143946 11266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:29.144954 11266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "7c3ffa2db25a49408d1410c08e4d997e"
format_stamp: "Formatted at 2026-06-18 23:24:29 on dist-test-slave-47qt"
server_key: "2a0ad05d64e06fb1cd0310df6c46850c"
server_key_iv: "29f3b6e183d123f2ee922491f6855b23"
server_key_version: "encryptionkey@0"
I20260618 23:24:29.145352 11266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "7c3ffa2db25a49408d1410c08e4d997e"
format_stamp: "Formatted at 2026-06-18 23:24:29 on dist-test-slave-47qt"
server_key: "2a0ad05d64e06fb1cd0310df6c46850c"
server_key_iv: "29f3b6e183d123f2ee922491f6855b23"
server_key_version: "encryptionkey@0"
I20260618 23:24:29.146906 11266 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:24:29.147626 11281 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:29.147822 11266 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:24:29.147926 11266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "7c3ffa2db25a49408d1410c08e4d997e"
format_stamp: "Formatted at 2026-06-18 23:24:29 on dist-test-slave-47qt"
server_key: "2a0ad05d64e06fb1cd0310df6c46850c"
server_key_iv: "29f3b6e183d123f2ee922491f6855b23"
server_key_version: "encryptionkey@0"
I20260618 23:24:29.148005 11266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:29.158217 11266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:29.160087 11266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:29.160250 11266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:29.164721 11266 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:42315
I20260618 23:24:29.164758 11333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:42315 every 8 connection(s)
I20260618 23:24:29.165108 11266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:24:29.165935 11334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:29.168054 11334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c3ffa2db25a49408d1410c08e4d997e: Bootstrap starting.
I20260618 23:24:29.168624 11334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7c3ffa2db25a49408d1410c08e4d997e: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:29.168917 11334 log.cc:824] T 00000000000000000000000000000000 P 7c3ffa2db25a49408d1410c08e4d997e: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:29.169603 11334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7c3ffa2db25a49408d1410c08e4d997e: No bootstrap required, opened a new log
I20260618 23:24:29.171096 11334 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7c3ffa2db25a49408d1410c08e4d997e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c3ffa2db25a49408d1410c08e4d997e" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 42315 } }
I20260618 23:24:29.171226 11334 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7c3ffa2db25a49408d1410c08e4d997e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:29.171247 11334 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7c3ffa2db25a49408d1410c08e4d997e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c3ffa2db25a49408d1410c08e4d997e, State: Initialized, Role: FOLLOWER
I20260618 23:24:29.171412 11334 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7c3ffa2db25a49408d1410c08e4d997e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c3ffa2db25a49408d1410c08e4d997e" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 42315 } }
I20260618 23:24:29.171490 11334 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7c3ffa2db25a49408d1410c08e4d997e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:24:29.171553 11334 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7c3ffa2db25a49408d1410c08e4d997e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:24:29.171602 11334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7c3ffa2db25a49408d1410c08e4d997e [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:29.172351 11334 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7c3ffa2db25a49408d1410c08e4d997e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c3ffa2db25a49408d1410c08e4d997e" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 42315 } }
I20260618 23:24:29.172441 11334 leader_election.cc:302] T 00000000000000000000000000000000 P 7c3ffa2db25a49408d1410c08e4d997e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7c3ffa2db25a49408d1410c08e4d997e; no voters:
I20260618 23:24:29.172572 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 11266
I20260618 23:24:29.172621 11334 leader_election.cc:288] T 00000000000000000000000000000000 P 7c3ffa2db25a49408d1410c08e4d997e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:24:29.172672 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:24:29.172659 11337 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7c3ffa2db25a49408d1410c08e4d997e [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:29.172811 11337 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7c3ffa2db25a49408d1410c08e4d997e [term 1 LEADER]: Becoming Leader. State: Replica: 7c3ffa2db25a49408d1410c08e4d997e, State: Running, Role: LEADER
I20260618 23:24:29.172866 11334 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7c3ffa2db25a49408d1410c08e4d997e [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:24:29.172878 8350 external_mini_cluster.cc:1428] Setting key 0020fa774eca459be7293af5466caf26
I20260618 23:24:29.172917 11337 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7c3ffa2db25a49408d1410c08e4d997e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c3ffa2db25a49408d1410c08e4d997e" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 42315 } }
I20260618 23:24:29.173280 11337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c3ffa2db25a49408d1410c08e4d997e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7c3ffa2db25a49408d1410c08e4d997e. Latest consensus state: current_term: 1 leader_uuid: "7c3ffa2db25a49408d1410c08e4d997e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c3ffa2db25a49408d1410c08e4d997e" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 42315 } } }
I20260618 23:24:29.173349 11337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c3ffa2db25a49408d1410c08e4d997e [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:29.173486 11338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7c3ffa2db25a49408d1410c08e4d997e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7c3ffa2db25a49408d1410c08e4d997e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7c3ffa2db25a49408d1410c08e4d997e" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 42315 } } }
I20260618 23:24:29.173543 11338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7c3ffa2db25a49408d1410c08e4d997e [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:29.173653 11344 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:24:29.174140 11344 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:24:29.178287 11344 catalog_manager.cc:1347] Generated new cluster ID: b7f56c76b8d147e69fb41ff644a89d2f
I20260618 23:24:29.178345 11344 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 18 23:24:29 dist-test-slave-47qt krb5kdc[11253](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:29.189275 11344 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:24:29.189898 11344 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:24:29.193334 11347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:29.173448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38538 (local address 127.8.39.190:42315)
0618 23:24:29.173792 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:29.173800 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:29.174363 (+ 563us) server_negotiation.cc:407] Connection header received
0618 23:24:29.174790 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:29.174798 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:29.174854 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:29.175000 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:29.175630 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:29.177022 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:29.178121 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:29.178743 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:29.178849 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:29.189650 (+ 10801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:29.189655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:29.189655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:29.189668 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:29.191674 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:29.192077 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:29.192080 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:29.192080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:29.192200 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:29.192400 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:29.192401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:29.192401 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:29.192673 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:29.192822 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:29.192921 (+ 99us) server_negotiation.cc:299] Negotiation successful
0618 23:24:29.193078 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":295,"thread_start_us":87,"threads_started":1}
I20260618 23:24:29.194224 8350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:24:29.197029 11344 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7c3ffa2db25a49408d1410c08e4d997e: Generated new TSK 0
I20260618 23:24:29.197211 11344 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:24:29.209228 11347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:29.204693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38548 (local address 127.8.39.190:42315)
0618 23:24:29.204763 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:29.204765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:29.204873 (+ 108us) server_negotiation.cc:407] Connection header received
0618 23:24:29.204943 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:29.204943 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:29.204953 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:29.204994 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:29.205545 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:29.206258 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:29.207578 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:29.207848 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:29.207947 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:29.208686 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:29.208688 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:29.208688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:24:29.208868 (+ 180us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:29.208889 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:24:29.208895 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:29.208945 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:29.209032 (+ 87us) server_negotiation.cc:299] Negotiation successful
0618 23:24:29.209068 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260618 23:24:29.211324 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 11266
2026-06-18T23:24:29Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1349 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:24:29 dist-test-slave-47qt krb5kdc[11367](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:24:29 dist-test-slave-47qt krb5kdc[11367](info): set up 2 sockets
Jun 18 23:24:29 dist-test-slave-47qt krb5kdc[11367](info): commencing operation
krb5kdc: starting...
W20260618 23:24:30.277697 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.049s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:24:30 dist-test-slave-47qt krb5kdc[11367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:24:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:24:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.39.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
I20260618 23:24:30.404268 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:46243
--webserver_interface=127.8.39.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:37669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:46243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:30.491783 11380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:30.492002 11380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:30.492024 11380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:30.493149 11380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:24:30.493189 11380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:30.493203 11380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:30.493216 11380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:24:30.493228 11380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:24:30.494619 11380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:37669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:46243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:46243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11380
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:30.494817 11380 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:30.494995 11380 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:30.500001 11388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:30.500005 11385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:30.500276 11386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:30.500290 11380 server_base.cc:1034] running on GCE node
I20260618 23:24:30.500571 11380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:30.500900 11380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:30.502064 11380 hybrid_clock.cc:648] HybridClock initialized: now 1781825070502058 us; error 38 us; skew 500 ppm
Jun 18 23:24:30 dist-test-slave-47qt krb5kdc[11367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825070, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:30.504438 11380 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:24:30.504881 11380 webserver.cc:460] Webserver started at http://127.8.39.190:43709/ using document root <none> and password file <none>
I20260618 23:24:30.505069 11380 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:30.505107 11380 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:30.505263 11380 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:30.506255 11380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "341b0fcc21984d93abc465bf719c553b"
format_stamp: "Formatted at 2026-06-18 23:24:30 on dist-test-slave-47qt"
server_key: "75c2f9150c26f7466683fa8455801f12"
server_key_iv: "2cb14fc4e6e3c60c2959b31a83d03c91"
server_key_version: "encryptionkey@0"
I20260618 23:24:30.506551 11380 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "341b0fcc21984d93abc465bf719c553b"
format_stamp: "Formatted at 2026-06-18 23:24:30 on dist-test-slave-47qt"
server_key: "75c2f9150c26f7466683fa8455801f12"
server_key_iv: "2cb14fc4e6e3c60c2959b31a83d03c91"
server_key_version: "encryptionkey@0"
I20260618 23:24:30.508020 11380 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260618 23:24:30.509418 11395 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:30.509722 11380 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:24:30.509828 11380 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "341b0fcc21984d93abc465bf719c553b"
format_stamp: "Formatted at 2026-06-18 23:24:30 on dist-test-slave-47qt"
server_key: "75c2f9150c26f7466683fa8455801f12"
server_key_iv: "2cb14fc4e6e3c60c2959b31a83d03c91"
server_key_version: "encryptionkey@0"
I20260618 23:24:30.509912 11380 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:30.518364 11380 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:30.520512 11380 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:30.520658 11380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:30.524900 11380 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:46243
I20260618 23:24:30.524928 11447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:46243 every 8 connection(s)
I20260618 23:24:30.525357 11380 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:24:30.526288 11448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:30.528486 11448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 341b0fcc21984d93abc465bf719c553b: Bootstrap starting.
I20260618 23:24:30.528981 11448 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 341b0fcc21984d93abc465bf719c553b: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:30.529299 11448 log.cc:824] T 00000000000000000000000000000000 P 341b0fcc21984d93abc465bf719c553b: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:30.529749 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 11380
I20260618 23:24:30.529877 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:24:30.530063 8350 external_mini_cluster.cc:1428] Setting key 5fe8d33f260cdd6c4ca9d0ae7faa3538
I20260618 23:24:30.530148 11448 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 341b0fcc21984d93abc465bf719c553b: No bootstrap required, opened a new log
I20260618 23:24:30.532044 11448 raft_consensus.cc:348] T 00000000000000000000000000000000 P 341b0fcc21984d93abc465bf719c553b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "341b0fcc21984d93abc465bf719c553b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 46243 } }
I20260618 23:24:30.532176 11448 raft_consensus.cc:374] T 00000000000000000000000000000000 P 341b0fcc21984d93abc465bf719c553b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:30.532197 11448 raft_consensus.cc:729] T 00000000000000000000000000000000 P 341b0fcc21984d93abc465bf719c553b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 341b0fcc21984d93abc465bf719c553b, State: Initialized, Role: FOLLOWER
I20260618 23:24:30.532342 11448 consensus_queue.cc:260] T 00000000000000000000000000000000 P 341b0fcc21984d93abc465bf719c553b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "341b0fcc21984d93abc465bf719c553b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 46243 } }
I20260618 23:24:30.532423 11448 raft_consensus.cc:388] T 00000000000000000000000000000000 P 341b0fcc21984d93abc465bf719c553b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:24:30.532445 11448 raft_consensus.cc:482] T 00000000000000000000000000000000 P 341b0fcc21984d93abc465bf719c553b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:24:30.532510 11448 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 341b0fcc21984d93abc465bf719c553b [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:30.533052 11448 raft_consensus.cc:504] T 00000000000000000000000000000000 P 341b0fcc21984d93abc465bf719c553b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "341b0fcc21984d93abc465bf719c553b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 46243 } }
I20260618 23:24:30.533205 11448 leader_election.cc:302] T 00000000000000000000000000000000 P 341b0fcc21984d93abc465bf719c553b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 341b0fcc21984d93abc465bf719c553b; no voters:
I20260618 23:24:30.533413 11448 leader_election.cc:288] T 00000000000000000000000000000000 P 341b0fcc21984d93abc465bf719c553b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:24:30.533437 11453 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 341b0fcc21984d93abc465bf719c553b [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:30.533555 11453 raft_consensus.cc:686] T 00000000000000000000000000000000 P 341b0fcc21984d93abc465bf719c553b [term 1 LEADER]: Becoming Leader. State: Replica: 341b0fcc21984d93abc465bf719c553b, State: Running, Role: LEADER
I20260618 23:24:30.533664 11448 sys_catalog.cc:564] T 00000000000000000000000000000000 P 341b0fcc21984d93abc465bf719c553b [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:24:30.533686 11453 consensus_queue.cc:237] T 00000000000000000000000000000000 P 341b0fcc21984d93abc465bf719c553b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "341b0fcc21984d93abc465bf719c553b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 46243 } }
I20260618 23:24:30.534034 11453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 341b0fcc21984d93abc465bf719c553b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 341b0fcc21984d93abc465bf719c553b. Latest consensus state: current_term: 1 leader_uuid: "341b0fcc21984d93abc465bf719c553b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "341b0fcc21984d93abc465bf719c553b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 46243 } } }
I20260618 23:24:30.534178 11453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 341b0fcc21984d93abc465bf719c553b [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:30.534050 11454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 341b0fcc21984d93abc465bf719c553b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "341b0fcc21984d93abc465bf719c553b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "341b0fcc21984d93abc465bf719c553b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 46243 } } }
I20260618 23:24:30.534736 11454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 341b0fcc21984d93abc465bf719c553b [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:30.534888 11462 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:24:30.535406 11462 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 18 23:24:30 dist-test-slave-47qt krb5kdc[11367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825070, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:30.538789 11462 catalog_manager.cc:1347] Generated new cluster ID: 1a988dddd18046588c040bf539dc73f4
I20260618 23:24:30.538844 11462 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:24:30.542138 11452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:30.530585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37128 (local address 127.8.39.190:46243)
0618 23:24:30.530747 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:30.530756 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:30.530833 (+ 77us) server_negotiation.cc:407] Connection header received
0618 23:24:30.531378 (+ 545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:30.531383 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:30.531426 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:30.531568 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:30.532441 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:30.533942 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:30.535283 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:30.535724 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:30.535855 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:30.538741 (+ 2886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:30.538744 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:30.538744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:30.538754 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:30.540462 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:30.540826 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:30.540829 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:30.540829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:30.540915 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:30.541186 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:30.541187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:30.541188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:30.541420 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:30.541582 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:30.541677 (+ 95us) server_negotiation.cc:299] Negotiation successful
0618 23:24:30.541983 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":85,"threads_started":1}
I20260618 23:24:30.585326 11462 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:24:30.586048 11462 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:24:30.595220 11462 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 341b0fcc21984d93abc465bf719c553b: Generated new TSK 0
I20260618 23:24:30.595682 11462 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.39.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
I20260618 23:24:30.628723 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.129:0
--local_ip_for_outbound_sockets=127.8.39.129
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:46243
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:37669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:30.713784 11474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:30.714001 11474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:30.714036 11474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:30.715224 11474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:30.715291 11474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:30.715338 11474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.129
I20260618 23:24:30.716737 11474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:37669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:46243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11474
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.129
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:30.716992 11474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:30.717240 11474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:30.721995 11480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:30.721961 11482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:30.721930 11479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:30.722158 11474 server_base.cc:1034] running on GCE node
I20260618 23:24:30.722364 11474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:30.722550 11474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:30.723706 11474 hybrid_clock.cc:648] HybridClock initialized: now 1781825070723688 us; error 25 us; skew 500 ppm
Jun 18 23:24:30 dist-test-slave-47qt krb5kdc[11367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825070, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:30.726166 11474 init.cc:377] Logged in from keytab as kudu/127.8.39.129@KRBTEST.COM (short username kudu)
I20260618 23:24:30.726630 11474 webserver.cc:460] Webserver started at http://127.8.39.129:38179/ using document root <none> and password file <none>
I20260618 23:24:30.726862 11474 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:30.726907 11474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:30.727041 11474 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:30.728000 11474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/data/instance:
uuid: "ac2afdcfe53e4db3ba97ec952506d1fe"
format_stamp: "Formatted at 2026-06-18 23:24:30 on dist-test-slave-47qt"
server_key: "24e28a5c1ca576855f435b451d62d01c"
server_key_iv: "77282cca0b649810197b884c91bba8a5"
server_key_version: "encryptionkey@0"
I20260618 23:24:30.728338 11474 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance:
uuid: "ac2afdcfe53e4db3ba97ec952506d1fe"
format_stamp: "Formatted at 2026-06-18 23:24:30 on dist-test-slave-47qt"
server_key: "24e28a5c1ca576855f435b451d62d01c"
server_key_iv: "77282cca0b649810197b884c91bba8a5"
server_key_version: "encryptionkey@0"
I20260618 23:24:30.729669 11474 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:24:30.730504 11489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:30.730693 11474 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:24:30.730795 11474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/wal
uuid: "ac2afdcfe53e4db3ba97ec952506d1fe"
format_stamp: "Formatted at 2026-06-18 23:24:30 on dist-test-slave-47qt"
server_key: "24e28a5c1ca576855f435b451d62d01c"
server_key_iv: "77282cca0b649810197b884c91bba8a5"
server_key_version: "encryptionkey@0"
I20260618 23:24:30.730880 11474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:30.756664 11474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:30.758638 11474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:30.758805 11474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:30.759037 11474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:30.759337 11474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:30.759389 11474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:30.759431 11474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:30.759459 11474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:30.765789 11474 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.129:46125
I20260618 23:24:30.765836 11602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.129:46125 every 8 connection(s)
I20260618 23:24:30.766144 11474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
Jun 18 23:24:30 dist-test-slave-47qt krb5kdc[11367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825070, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:30.774183 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 11474
I20260618 23:24:30.774293 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance
I20260618 23:24:30.774499 8350 external_mini_cluster.cc:1428] Setting key 0ec8a076368f5caf7569716f3748fa36
I20260618 23:24:30.775378 11452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:30.766904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:39485 (local address 127.8.39.190:46243)
0618 23:24:30.766974 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:30.766976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:30.767130 (+ 154us) server_negotiation.cc:407] Connection header received
0618 23:24:30.767563 (+ 433us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:30.767565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:30.767575 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:30.767639 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:30.768556 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:30.769057 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:30.770056 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:30.770262 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:30.770335 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:30.772570 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:30.772572 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:30.772572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:30.772578 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:30.773918 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:30.774416 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:30.774417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:30.774418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:30.774472 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:30.774752 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:30.774753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:30.774753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:30.774927 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:30.774994 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:30.775236 (+ 242us) server_negotiation.cc:299] Negotiation successful
0618 23:24:30.775313 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260618 23:24:30.775676 11605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:30.766820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:46243 (local address 127.8.39.129:39485)
0618 23:24:30.766969 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:30.767021 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:30.767480 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:30.767689 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:30.767692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:30.768237 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:30.768424 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:30.768428 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:30.769206 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:30.769207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:30.769979 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:30.769980 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:30.770375 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:30.770376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:30.770480 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:30.770948 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:30.770956 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:30.772484 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:30.774007 (+ 1523us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:30.774014 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:30.774016 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:30.774331 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:30.774542 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:30.774543 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:30.774543 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:30.774676 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:30.774992 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:30.774997 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:30.775104 (+ 107us) client_negotiation.cc:769] Sending connection context
0618 23:24:30.775217 (+ 113us) client_negotiation.cc:240] Negotiation successful
0618 23:24:30.775405 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":85,"threads_started":1}
I20260618 23:24:30.776893 11603 heartbeater.cc:347] Connected to a master server at 127.8.39.190:46243
I20260618 23:24:30.776969 11603 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:30.777200 11603 heartbeater.cc:511] Master 127.8.39.190:46243 requested a full tablet report, sending...
I20260618 23:24:30.777700 11412 ts_manager.cc:194] Registered new tserver with Master: ac2afdcfe53e4db3ba97ec952506d1fe (127.8.39.129:46125)
I20260618 23:24:30.778498 11412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.129@KRBTEST.COM'} at 127.8.39.129:39485
WARNING: no policy specified for kudu/127.8.39.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
I20260618 23:24:30.800578 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.130:0
--local_ip_for_outbound_sockets=127.8.39.130
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:46243
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:37669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:30.879760 11608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:30.880016 11608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:30.880057 11608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:30.881228 11608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:30.881305 11608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:30.881353 11608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.130
I20260618 23:24:30.882828 11608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:37669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:46243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.130
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:30.883100 11608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:30.883335 11608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:30.888252 11616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:30.888734 11614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:30.888767 11613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:30.888799 11608 server_base.cc:1034] running on GCE node
I20260618 23:24:30.889024 11608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:30.889330 11608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:30.890601 11608 hybrid_clock.cc:648] HybridClock initialized: now 1781825070890583 us; error 37 us; skew 500 ppm
Jun 18 23:24:30 dist-test-slave-47qt krb5kdc[11367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825070, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:30.893493 11608 init.cc:377] Logged in from keytab as kudu/127.8.39.130@KRBTEST.COM (short username kudu)
I20260618 23:24:30.893985 11608 webserver.cc:460] Webserver started at http://127.8.39.130:42829/ using document root <none> and password file <none>
I20260618 23:24:30.894203 11608 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:30.894253 11608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:30.894358 11608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:30.895467 11608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/data/instance:
uuid: "afef013beaf7427d9da7ea9317d18d77"
format_stamp: "Formatted at 2026-06-18 23:24:30 on dist-test-slave-47qt"
server_key: "3ccfc064b1b1f2d35bddf81cdcf0e126"
server_key_iv: "a58f87f2371076346a48081f8f4af0ec"
server_key_version: "encryptionkey@0"
I20260618 23:24:30.895784 11608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance:
uuid: "afef013beaf7427d9da7ea9317d18d77"
format_stamp: "Formatted at 2026-06-18 23:24:30 on dist-test-slave-47qt"
server_key: "3ccfc064b1b1f2d35bddf81cdcf0e126"
server_key_iv: "a58f87f2371076346a48081f8f4af0ec"
server_key_version: "encryptionkey@0"
I20260618 23:24:30.897167 11608 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:24:30.898319 11623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:30.898713 11608 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:30.898864 11608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/wal
uuid: "afef013beaf7427d9da7ea9317d18d77"
format_stamp: "Formatted at 2026-06-18 23:24:30 on dist-test-slave-47qt"
server_key: "3ccfc064b1b1f2d35bddf81cdcf0e126"
server_key_iv: "a58f87f2371076346a48081f8f4af0ec"
server_key_version: "encryptionkey@0"
I20260618 23:24:30.898947 11608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:30.912642 11608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:30.915203 11608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:30.915369 11608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:30.915596 11608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:30.915897 11608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:30.915949 11608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:30.915992 11608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:30.916020 11608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:30.922240 11608 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.130:35383
I20260618 23:24:30.922351 11736 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.130:35383 every 8 connection(s)
I20260618 23:24:30.922662 11608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
I20260618 23:24:30.925345 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 11608
I20260618 23:24:30.925464 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance
I20260618 23:24:30.925662 8350 external_mini_cluster.cc:1428] Setting key 16e5ea4e9b9bd8f971f7d236f6dacb0c
Jun 18 23:24:30 dist-test-slave-47qt krb5kdc[11367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825070, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:30.933514 11452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:30.923557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:53735 (local address 127.8.39.190:46243)
0618 23:24:30.923642 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:30.923644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:30.923831 (+ 187us) server_negotiation.cc:407] Connection header received
0618 23:24:30.924340 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:30.924341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:30.924361 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:30.924420 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:30.925573 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:30.926282 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:30.927858 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:30.928157 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:30.928194 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:30.930617 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:30.930619 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:30.930619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:30.930627 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:30.932029 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:30.932435 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:30.932437 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:30.932437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:30.932494 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:30.932752 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:30.932753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:30.932753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:30.932873 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:30.932947 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:30.933339 (+ 392us) server_negotiation.cc:299] Negotiation successful
0618 23:24:30.933417 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 23:24:30.933763 11739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:30.923499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:46243 (local address 127.8.39.130:53735)
0618 23:24:30.923702 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:30.923738 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:30.924218 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:30.924587 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:30.924590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:30.925254 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:30.925473 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:30.925478 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:30.926391 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:30.926392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:30.927746 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:30.927749 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:30.928269 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:30.928269 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:30.928380 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:30.928987 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:30.928998 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:30.930517 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:30.932127 (+ 1610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:30.932134 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:30.932136 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:30.932357 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:30.932559 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:30.932560 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:30.932560 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:30.932673 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:30.932938 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:30.932944 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:30.933184 (+ 240us) client_negotiation.cc:769] Sending connection context
0618 23:24:30.933325 (+ 141us) client_negotiation.cc:240] Negotiation successful
0618 23:24:30.933484 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":128,"threads_started":1}
WARNING: no policy specified for kudu/127.8.39.131@KRBTEST.COM; defaulting to no policy
I20260618 23:24:30.934424 11737 heartbeater.cc:347] Connected to a master server at 127.8.39.190:46243
I20260618 23:24:30.934530 11737 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:30.934703 11737 heartbeater.cc:511] Master 127.8.39.190:46243 requested a full tablet report, sending...
I20260618 23:24:30.935122 11412 ts_manager.cc:194] Registered new tserver with Master: afef013beaf7427d9da7ea9317d18d77 (127.8.39.130:35383)
I20260618 23:24:30.935546 11412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.130@KRBTEST.COM'} at 127.8.39.130:53735
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
I20260618 23:24:30.953038 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.131:0
--local_ip_for_outbound_sockets=127.8.39.131
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:46243
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:37669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:31.037883 11742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:31.038093 11742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:31.038118 11742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:31.039264 11742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:31.039305 11742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:31.039357 11742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.131
I20260618 23:24:31.040781 11742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:37669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:46243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.131
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:31.041024 11742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:31.041299 11742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:31.046723 11748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:31.046723 11747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:31.046903 11750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:31.047211 11742 server_base.cc:1034] running on GCE node
I20260618 23:24:31.047456 11742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:31.047698 11742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:31.048892 11742 hybrid_clock.cc:648] HybridClock initialized: now 1781825071048882 us; error 32 us; skew 500 ppm
Jun 18 23:24:31 dist-test-slave-47qt krb5kdc[11367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825071, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:31.051762 11742 init.cc:377] Logged in from keytab as kudu/127.8.39.131@KRBTEST.COM (short username kudu)
I20260618 23:24:31.052320 11742 webserver.cc:460] Webserver started at http://127.8.39.131:46391/ using document root <none> and password file <none>
I20260618 23:24:31.052574 11742 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:31.052650 11742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:31.052824 11742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:31.054040 11742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/data/instance:
uuid: "7461f64f5e9b4209bf2485e6fd194dc0"
format_stamp: "Formatted at 2026-06-18 23:24:31 on dist-test-slave-47qt"
server_key: "bdc777fc7588e53fe40e73b0f10f2674"
server_key_iv: "9c4394ef7ef695c26e4d566bb7a22823"
server_key_version: "encryptionkey@0"
I20260618 23:24:31.054435 11742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance:
uuid: "7461f64f5e9b4209bf2485e6fd194dc0"
format_stamp: "Formatted at 2026-06-18 23:24:31 on dist-test-slave-47qt"
server_key: "bdc777fc7588e53fe40e73b0f10f2674"
server_key_iv: "9c4394ef7ef695c26e4d566bb7a22823"
server_key_version: "encryptionkey@0"
I20260618 23:24:31.055893 11742 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260618 23:24:31.056788 11757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:31.057029 11742 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:24:31.057252 11742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/wal
uuid: "7461f64f5e9b4209bf2485e6fd194dc0"
format_stamp: "Formatted at 2026-06-18 23:24:31 on dist-test-slave-47qt"
server_key: "bdc777fc7588e53fe40e73b0f10f2674"
server_key_iv: "9c4394ef7ef695c26e4d566bb7a22823"
server_key_version: "encryptionkey@0"
I20260618 23:24:31.057384 11742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:31.067282 11742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:31.069835 11742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:31.070003 11742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:31.070255 11742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:31.070561 11742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:31.070616 11742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:31.070660 11742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:31.070683 11742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:31.076615 11742 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.131:38723
I20260618 23:24:31.076659 11870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.131:38723 every 8 connection(s)
I20260618 23:24:31.076967 11742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
I20260618 23:24:31.078018 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 11742
I20260618 23:24:31.078130 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance
I20260618 23:24:31.078329 8350 external_mini_cluster.cc:1428] Setting key 97ed5dd65fa2cf15ce24599adb250c5e
Jun 18 23:24:31 dist-test-slave-47qt krb5kdc[11367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825071, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:31.087457 11452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:31.077713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:45053 (local address 127.8.39.190:46243)
0618 23:24:31.077796 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:31.077798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:31.078059 (+ 261us) server_negotiation.cc:407] Connection header received
0618 23:24:31.078626 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:31.078628 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:31.078640 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:31.078743 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:31.079789 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:31.080319 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:31.081664 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:31.081947 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:31.082012 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:31.084793 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:31.084794 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:31.084794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:31.084802 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:31.086209 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:31.086622 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:31.086623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:31.086623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:31.086671 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:31.086901 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:31.086902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:31.086902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:31.087008 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:31.087061 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:31.087326 (+ 265us) server_negotiation.cc:299] Negotiation successful
0618 23:24:31.087392 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260618 23:24:31.087630 11873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:31.077694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:46243 (local address 127.8.39.131:45053)
0618 23:24:31.077909 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:31.077950 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:31.078517 (+ 567us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:31.078835 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:31.078841 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:31.079516 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:31.079711 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:31.079716 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:31.080516 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:31.080517 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:31.081524 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:31.081526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:31.082054 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:31.082055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:31.082173 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:31.082803 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:31.082812 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:31.084650 (+ 1838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:31.086302 (+ 1652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:31.086308 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:31.086310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:31.086549 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:31.086732 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:31.086732 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:31.086733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:31.086836 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:31.087068 (+ 232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:31.087073 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:31.087203 (+ 130us) client_negotiation.cc:769] Sending connection context
0618 23:24:31.087293 (+ 90us) client_negotiation.cc:240] Negotiation successful
0618 23:24:31.087418 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":118,"threads_started":1}
I20260618 23:24:31.088346 11871 heartbeater.cc:347] Connected to a master server at 127.8.39.190:46243
I20260618 23:24:31.088438 11871 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:31.088589 11871 heartbeater.cc:511] Master 127.8.39.190:46243 requested a full tablet report, sending...
I20260618 23:24:31.088971 11412 ts_manager.cc:194] Registered new tserver with Master: 7461f64f5e9b4209bf2485e6fd194dc0 (127.8.39.131:38723)
I20260618 23:24:31.089421 11412 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.131@KRBTEST.COM'} at 127.8.39.131:45053
I20260618 23:24:31.090294 8350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 18 23:24:31 dist-test-slave-47qt krb5kdc[11367](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825071, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 18 23:24:31 dist-test-slave-47qt krb5kdc[11367](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825071, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:31.116062 11452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:31.108334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37132 (local address 127.8.39.190:46243)
0618 23:24:31.108399 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:31.108400 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:31.108447 (+ 47us) server_negotiation.cc:407] Connection header received
0618 23:24:31.108494 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:31.108495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:31.108504 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:31.108564 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:31.109239 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:31.109715 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:31.110535 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:31.110714 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:31.110803 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:31.113622 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:31.113623 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:31.113623 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:31.113633 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:31.114894 (+ 1261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:31.115244 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:31.115245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:31.115245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:31.115319 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:31.115519 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:31.115520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:31.115520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:31.115670 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:31.115775 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:31.115887 (+ 112us) server_negotiation.cc:299] Negotiation successful
0618 23:24:31.115960 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
W20260618 23:24:31.116390 11412 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:37132
I20260618 23:24:31.116871 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 11474
I20260618 23:24:31.121464 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 11608
I20260618 23:24:31.125821 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 11742
I20260618 23:24:31.130263 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 11380
2026-06-18T23:24:31Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1921 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:24:31 dist-test-slave-47qt krb5kdc[11883](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:24:31 dist-test-slave-47qt krb5kdc[11883](info): set up 2 sockets
Jun 18 23:24:31 dist-test-slave-47qt krb5kdc[11883](info): commencing operation
krb5kdc: starting...
W20260618 23:24:32.181463 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:24:32 dist-test-slave-47qt krb5kdc[11883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:24:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:24:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.39.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
I20260618 23:24:32.300144 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:39825
--webserver_interface=127.8.39.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:40163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:39825
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:32.379499 11896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:32.379689 11896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:32.379711 11896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:32.380820 11896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:24:32.380860 11896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:32.380875 11896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:32.380887 11896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:24:32.380937 11896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:24:32.382442 11896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:40163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:39825
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:39825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:32.382686 11896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:32.382910 11896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:32.387809 11904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:32.387809 11901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:32.387833 11902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:32.387953 11896 server_base.cc:1034] running on GCE node
I20260618 23:24:32.388509 11896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:32.388765 11896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:32.389917 11896 hybrid_clock.cc:648] HybridClock initialized: now 1781825072389866 us; error 73 us; skew 500 ppm
Jun 18 23:24:32 dist-test-slave-47qt krb5kdc[11883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825072, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:32.392846 11896 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:24:32.393316 11896 webserver.cc:460] Webserver started at http://127.8.39.190:44269/ using document root <none> and password file <none>
I20260618 23:24:32.393539 11896 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:32.393605 11896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:32.393752 11896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:32.394841 11896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "da94c796bc5a4441a53d5070bbfe3aa0"
format_stamp: "Formatted at 2026-06-18 23:24:32 on dist-test-slave-47qt"
server_key: "2f5e3f6d77a739829b48175f1ec31698"
server_key_iv: "034ade2cf6b9f026f15fa042dcaa73df"
server_key_version: "encryptionkey@0"
I20260618 23:24:32.395195 11896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "da94c796bc5a4441a53d5070bbfe3aa0"
format_stamp: "Formatted at 2026-06-18 23:24:32 on dist-test-slave-47qt"
server_key: "2f5e3f6d77a739829b48175f1ec31698"
server_key_iv: "034ade2cf6b9f026f15fa042dcaa73df"
server_key_version: "encryptionkey@0"
I20260618 23:24:32.396471 11896 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:24:32.397238 11911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:32.397408 11896 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:32.397502 11896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "da94c796bc5a4441a53d5070bbfe3aa0"
format_stamp: "Formatted at 2026-06-18 23:24:32 on dist-test-slave-47qt"
server_key: "2f5e3f6d77a739829b48175f1ec31698"
server_key_iv: "034ade2cf6b9f026f15fa042dcaa73df"
server_key_version: "encryptionkey@0"
I20260618 23:24:32.397586 11896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:32.434890 11896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:32.437059 11896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:32.437268 11896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:32.440827 11896 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:39825
I20260618 23:24:32.440871 11963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:39825 every 8 connection(s)
I20260618 23:24:32.441373 11896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:24:32.442019 11964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:32.444612 11964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da94c796bc5a4441a53d5070bbfe3aa0: Bootstrap starting.
I20260618 23:24:32.444888 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 11896
I20260618 23:24:32.445003 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:24:32.445242 8350 external_mini_cluster.cc:1428] Setting key 057415475d8d13a8b1623d7534e93cb2
I20260618 23:24:32.445374 11964 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da94c796bc5a4441a53d5070bbfe3aa0: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:32.445739 11964 log.cc:824] T 00000000000000000000000000000000 P da94c796bc5a4441a53d5070bbfe3aa0: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:32.446439 11964 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da94c796bc5a4441a53d5070bbfe3aa0: No bootstrap required, opened a new log
I20260618 23:24:32.448237 11964 raft_consensus.cc:348] T 00000000000000000000000000000000 P da94c796bc5a4441a53d5070bbfe3aa0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da94c796bc5a4441a53d5070bbfe3aa0" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 39825 } }
I20260618 23:24:32.448369 11964 raft_consensus.cc:374] T 00000000000000000000000000000000 P da94c796bc5a4441a53d5070bbfe3aa0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:32.448392 11964 raft_consensus.cc:729] T 00000000000000000000000000000000 P da94c796bc5a4441a53d5070bbfe3aa0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da94c796bc5a4441a53d5070bbfe3aa0, State: Initialized, Role: FOLLOWER
I20260618 23:24:32.448515 11964 consensus_queue.cc:260] T 00000000000000000000000000000000 P da94c796bc5a4441a53d5070bbfe3aa0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da94c796bc5a4441a53d5070bbfe3aa0" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 39825 } }
I20260618 23:24:32.448585 11964 raft_consensus.cc:388] T 00000000000000000000000000000000 P da94c796bc5a4441a53d5070bbfe3aa0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:24:32.448633 11964 raft_consensus.cc:482] T 00000000000000000000000000000000 P da94c796bc5a4441a53d5070bbfe3aa0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:24:32.448679 11964 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da94c796bc5a4441a53d5070bbfe3aa0 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:32.449256 11964 raft_consensus.cc:504] T 00000000000000000000000000000000 P da94c796bc5a4441a53d5070bbfe3aa0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da94c796bc5a4441a53d5070bbfe3aa0" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 39825 } }
I20260618 23:24:32.449365 11964 leader_election.cc:302] T 00000000000000000000000000000000 P da94c796bc5a4441a53d5070bbfe3aa0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da94c796bc5a4441a53d5070bbfe3aa0; no voters:
I20260618 23:24:32.449530 11964 leader_election.cc:288] T 00000000000000000000000000000000 P da94c796bc5a4441a53d5070bbfe3aa0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:24:32.449580 11969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da94c796bc5a4441a53d5070bbfe3aa0 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:32.449688 11969 raft_consensus.cc:686] T 00000000000000000000000000000000 P da94c796bc5a4441a53d5070bbfe3aa0 [term 1 LEADER]: Becoming Leader. State: Replica: da94c796bc5a4441a53d5070bbfe3aa0, State: Running, Role: LEADER
I20260618 23:24:32.449769 11964 sys_catalog.cc:564] T 00000000000000000000000000000000 P da94c796bc5a4441a53d5070bbfe3aa0 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:24:32.449761 11969 consensus_queue.cc:237] T 00000000000000000000000000000000 P da94c796bc5a4441a53d5070bbfe3aa0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da94c796bc5a4441a53d5070bbfe3aa0" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 39825 } }
I20260618 23:24:32.451696 11970 sys_catalog.cc:455] T 00000000000000000000000000000000 P da94c796bc5a4441a53d5070bbfe3aa0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader da94c796bc5a4441a53d5070bbfe3aa0. Latest consensus state: current_term: 1 leader_uuid: "da94c796bc5a4441a53d5070bbfe3aa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da94c796bc5a4441a53d5070bbfe3aa0" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 39825 } } }
I20260618 23:24:32.451789 11970 sys_catalog.cc:458] T 00000000000000000000000000000000 P da94c796bc5a4441a53d5070bbfe3aa0 [sys.catalog]: This master's current role is: LEADER
W20260618 23:24:32.451980 11983 catalog_manager.cc:1558] T 00000000000000000000000000000000 P da94c796bc5a4441a53d5070bbfe3aa0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260618 23:24:32.452020 11971 sys_catalog.cc:455] T 00000000000000000000000000000000 P da94c796bc5a4441a53d5070bbfe3aa0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da94c796bc5a4441a53d5070bbfe3aa0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da94c796bc5a4441a53d5070bbfe3aa0" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 39825 } } }
W20260618 23:24:32.452168 11983 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:24:32.452183 11971 sys_catalog.cc:458] T 00000000000000000000000000000000 P da94c796bc5a4441a53d5070bbfe3aa0 [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:32.452560 11986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:24:32.452760 11986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 18 23:24:32 dist-test-slave-47qt krb5kdc[11883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:32.454521 11986 catalog_manager.cc:1347] Generated new cluster ID: f743f84d74cc48d3b68aaa16516c2ff6
I20260618 23:24:32.454579 11986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:24:32.457337 11967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:32.445849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32852 (local address 127.8.39.190:39825)
0618 23:24:32.446128 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:32.446137 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:32.446169 (+ 32us) server_negotiation.cc:407] Connection header received
0618 23:24:32.446702 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:32.446709 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:32.446756 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:32.446912 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:32.447712 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:32.449158 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:32.450162 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:32.450657 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:32.450773 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:32.453667 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:32.453671 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:32.453671 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:32.453683 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:32.455585 (+ 1902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:32.455989 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:32.455992 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:32.455992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:32.456084 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:32.456293 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:32.456294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:32.456294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:32.456564 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:32.456704 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:32.456823 (+ 119us) server_negotiation.cc:299] Negotiation successful
0618 23:24:32.456941 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":96,"threads_started":1}
I20260618 23:24:32.467597 11986 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:24:32.468590 11986 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:24:32.477412 11986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da94c796bc5a4441a53d5070bbfe3aa0: Generated new TSK 0
I20260618 23:24:32.477577 11986 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.39.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
I20260618 23:24:32.504869 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.129:0
--local_ip_for_outbound_sockets=127.8.39.129
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:39825
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:40163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:32.586650 11990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:32.586840 11990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:32.586863 11990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:32.588056 11990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:32.588114 11990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:32.588145 11990 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.129
I20260618 23:24:32.589780 11990 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:40163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:39825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.129
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:32.590009 11990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:32.590227 11990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:32.595170 11998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:32.595170 11995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:32.595191 11996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:32.595202 11990 server_base.cc:1034] running on GCE node
I20260618 23:24:32.595564 11990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:32.595759 11990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:32.596980 11990 hybrid_clock.cc:648] HybridClock initialized: now 1781825072596968 us; error 27 us; skew 500 ppm
Jun 18 23:24:32 dist-test-slave-47qt krb5kdc[11883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825072, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:32.599505 11990 init.cc:377] Logged in from keytab as kudu/127.8.39.129@KRBTEST.COM (short username kudu)
I20260618 23:24:32.599858 11990 webserver.cc:460] Webserver started at http://127.8.39.129:38761/ using document root <none> and password file <none>
I20260618 23:24:32.600098 11990 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:32.600142 11990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:32.600270 11990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:32.601328 11990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-0/data/instance:
uuid: "1b1b7e8854c844adaf3033230ebf8e12"
format_stamp: "Formatted at 2026-06-18 23:24:32 on dist-test-slave-47qt"
server_key: "f832aacb228085168ed2d7ec076659d1"
server_key_iv: "25832209936b55cbff95cbcf0904a137"
server_key_version: "encryptionkey@0"
I20260618 23:24:32.601667 11990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance:
uuid: "1b1b7e8854c844adaf3033230ebf8e12"
format_stamp: "Formatted at 2026-06-18 23:24:32 on dist-test-slave-47qt"
server_key: "f832aacb228085168ed2d7ec076659d1"
server_key_iv: "25832209936b55cbff95cbcf0904a137"
server_key_version: "encryptionkey@0"
I20260618 23:24:32.602977 11990 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:24:32.603716 12005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:32.603909 11990 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:32.604002 11990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-0/wal
uuid: "1b1b7e8854c844adaf3033230ebf8e12"
format_stamp: "Formatted at 2026-06-18 23:24:32 on dist-test-slave-47qt"
server_key: "f832aacb228085168ed2d7ec076659d1"
server_key_iv: "25832209936b55cbff95cbcf0904a137"
server_key_version: "encryptionkey@0"
I20260618 23:24:32.604080 11990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:32.612165 11990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:32.614454 11990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:32.614609 11990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:32.614840 11990 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:32.615135 11990 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:32.615185 11990 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:32.615224 11990 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:32.615249 11990 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:32.620884 11990 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.129:42395
I20260618 23:24:32.620956 12118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.129:42395 every 8 connection(s)
I20260618 23:24:32.621320 11990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
Jun 18 23:24:32 dist-test-slave-47qt krb5kdc[11883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825072, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:32.630479 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 11990
I20260618 23:24:32.630587 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance
I20260618 23:24:32.630787 8350 external_mini_cluster.cc:1428] Setting key d21880e108aaaf3ca4f8fdc62d4c73fb
I20260618 23:24:32.631551 11967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:32.622111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:56645 (local address 127.8.39.190:39825)
0618 23:24:32.622160 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:32.622162 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:32.622273 (+ 111us) server_negotiation.cc:407] Connection header received
0618 23:24:32.623186 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:32.623187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:32.623198 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:32.623262 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:32.623850 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:32.624464 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:32.625599 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:32.625787 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:32.625844 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:32.627886 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:32.627887 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:32.627887 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:32.627893 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:32.629786 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:32.630290 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:32.630292 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:32.630292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:32.630352 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:32.630703 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:32.630704 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:32.630704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:32.630813 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:32.630938 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:32.631341 (+ 403us) server_negotiation.cc:299] Negotiation successful
0618 23:24:32.631468 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260618 23:24:32.631738 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:32.622016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:39825 (local address 127.8.39.129:56645)
0618 23:24:32.622166 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:32.622205 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:32.623069 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:32.623314 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:32.623317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:32.623554 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:32.623736 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:32.623740 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:32.624577 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:32.624578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:32.625507 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:32.625510 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:32.625863 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:32.625864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:32.625989 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:32.626440 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:32.626448 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:32.627793 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:32.629931 (+ 2138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:32.629938 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:32.629940 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:32.630185 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:32.630436 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:32.630437 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:32.630437 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:32.630596 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:32.630900 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:32.630907 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:32.631172 (+ 265us) client_negotiation.cc:769] Sending connection context
0618 23:24:32.631295 (+ 123us) client_negotiation.cc:240] Negotiation successful
0618 23:24:32.631488 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260618 23:24:32.632280 12119 heartbeater.cc:347] Connected to a master server at 127.8.39.190:39825
I20260618 23:24:32.632393 12119 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:32.632560 12119 heartbeater.cc:511] Master 127.8.39.190:39825 requested a full tablet report, sending...
I20260618 23:24:32.633193 11928 ts_manager.cc:194] Registered new tserver with Master: 1b1b7e8854c844adaf3033230ebf8e12 (127.8.39.129:42395)
I20260618 23:24:32.634112 11928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.129@KRBTEST.COM'} at 127.8.39.129:56645
WARNING: no policy specified for kudu/127.8.39.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
I20260618 23:24:32.656416 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.130:0
--local_ip_for_outbound_sockets=127.8.39.130
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:39825
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:40163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:32.736773 12124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:32.736968 12124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:32.736992 12124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:32.738160 12124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:32.738201 12124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:32.738229 12124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.130
I20260618 23:24:32.739619 12124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:40163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:39825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.130
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:32.739786 12124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:32.739949 12124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:32.744879 12129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:32.744884 12130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:32.744884 12132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:32.745281 12124 server_base.cc:1034] running on GCE node
I20260618 23:24:32.745522 12124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:32.745754 12124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:32.746955 12124 hybrid_clock.cc:648] HybridClock initialized: now 1781825072746941 us; error 60 us; skew 500 ppm
Jun 18 23:24:32 dist-test-slave-47qt krb5kdc[11883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825072, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:32.750111 12124 init.cc:377] Logged in from keytab as kudu/127.8.39.130@KRBTEST.COM (short username kudu)
I20260618 23:24:32.750620 12124 webserver.cc:460] Webserver started at http://127.8.39.130:38755/ using document root <none> and password file <none>
I20260618 23:24:32.750797 12124 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:32.750836 12124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:32.750922 12124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:32.751940 12124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-1/data/instance:
uuid: "dc0e05c064974b4cb43c35cdf0eb4b4e"
format_stamp: "Formatted at 2026-06-18 23:24:32 on dist-test-slave-47qt"
server_key: "1d0973c432c25c4772314f37230e19f4"
server_key_iv: "236df09554e348fa9c11fe36f8dbb2c1"
server_key_version: "encryptionkey@0"
I20260618 23:24:32.752242 12124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance:
uuid: "dc0e05c064974b4cb43c35cdf0eb4b4e"
format_stamp: "Formatted at 2026-06-18 23:24:32 on dist-test-slave-47qt"
server_key: "1d0973c432c25c4772314f37230e19f4"
server_key_iv: "236df09554e348fa9c11fe36f8dbb2c1"
server_key_version: "encryptionkey@0"
I20260618 23:24:32.753536 12124 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260618 23:24:32.754199 12139 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:32.754391 12124 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260618 23:24:32.754496 12124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-1/wal
uuid: "dc0e05c064974b4cb43c35cdf0eb4b4e"
format_stamp: "Formatted at 2026-06-18 23:24:32 on dist-test-slave-47qt"
server_key: "1d0973c432c25c4772314f37230e19f4"
server_key_iv: "236df09554e348fa9c11fe36f8dbb2c1"
server_key_version: "encryptionkey@0"
I20260618 23:24:32.754585 12124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:32.766796 12124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:32.768977 12124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:32.769205 12124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:32.769482 12124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:32.769789 12124 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:32.769845 12124 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:32.769882 12124 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:32.769908 12124 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:32.775619 12124 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.130:45023
I20260618 23:24:32.775691 12252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.130:45023 every 8 connection(s)
I20260618 23:24:32.776000 12124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
I20260618 23:24:32.780982 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 12124
I20260618 23:24:32.781118 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance
I20260618 23:24:32.781317 8350 external_mini_cluster.cc:1428] Setting key 372359ee18e8766d581b651d092433de
Jun 18 23:24:32 dist-test-slave-47qt krb5kdc[11883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825072, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:32.786078 11967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:32.776688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:52131 (local address 127.8.39.190:39825)
0618 23:24:32.776770 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:32.776772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:32.776977 (+ 205us) server_negotiation.cc:407] Connection header received
0618 23:24:32.777926 (+ 949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:32.777927 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:32.777936 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:32.777988 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:32.778556 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:32.779032 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:32.779979 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:32.780167 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:32.780223 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:32.782635 (+ 2412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:32.782636 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:32.782637 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:32.782644 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:32.784521 (+ 1877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:32.785030 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:32.785031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:32.785031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:32.785094 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:32.785421 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:32.785422 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:32.785422 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:32.785545 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:32.785625 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:32.785896 (+ 271us) server_negotiation.cc:299] Negotiation successful
0618 23:24:32.785974 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 23:24:32.786271 12255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:32.776646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:39825 (local address 127.8.39.130:52131)
0618 23:24:32.776854 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:32.776888 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:32.777834 (+ 946us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:32.778039 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:32.778042 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:32.778301 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:32.778478 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:32.778481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:32.779111 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:32.779112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:32.779901 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:32.779903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:32.780241 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:32.780241 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:32.780354 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:32.780868 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:32.780876 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:32.782531 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:32.784624 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:32.784630 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:32.784633 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:32.784964 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:32.785232 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:32.785233 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:32.785234 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:32.785349 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:32.785616 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:32.785622 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:32.785763 (+ 141us) client_negotiation.cc:769] Sending connection context
0618 23:24:32.785873 (+ 110us) client_negotiation.cc:240] Negotiation successful
0618 23:24:32.786031 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":101,"threads_started":1}
I20260618 23:24:32.786782 12253 heartbeater.cc:347] Connected to a master server at 127.8.39.190:39825
I20260618 23:24:32.786880 12253 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:32.787216 12253 heartbeater.cc:511] Master 127.8.39.190:39825 requested a full tablet report, sending...
I20260618 23:24:32.787657 11928 ts_manager.cc:194] Registered new tserver with Master: dc0e05c064974b4cb43c35cdf0eb4b4e (127.8.39.130:45023)
I20260618 23:24:32.788164 11928 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.130@KRBTEST.COM'} at 127.8.39.130:52131
WARNING: no policy specified for kudu/127.8.39.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
I20260618 23:24:32.811661 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.131:0
--local_ip_for_outbound_sockets=127.8.39.131
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:39825
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:40163
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:32.891242 12258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:32.891439 12258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:32.891461 12258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:32.892521 12258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:32.892555 12258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:32.892582 12258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.131
I20260618 23:24:32.894145 12258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:40163
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:39825
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.131
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:32.894316 12258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:32.894479 12258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:32.899338 12266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:32.899345 12264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:32.899422 12258 server_base.cc:1034] running on GCE node
W20260618 23:24:32.899346 12263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:32.899781 12258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:32.900018 12258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:32.901292 12258 hybrid_clock.cc:648] HybridClock initialized: now 1781825072901281 us; error 28 us; skew 500 ppm
Jun 18 23:24:32 dist-test-slave-47qt krb5kdc[11883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825072, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:32.903678 12258 init.cc:377] Logged in from keytab as kudu/127.8.39.131@KRBTEST.COM (short username kudu)
I20260618 23:24:32.904199 12258 webserver.cc:460] Webserver started at http://127.8.39.131:42131/ using document root <none> and password file <none>
I20260618 23:24:32.904419 12258 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:32.904485 12258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:32.904639 12258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:32.905711 12258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-2/data/instance:
uuid: "ede25656177d49d694a7d6ba5f38b308"
format_stamp: "Formatted at 2026-06-18 23:24:32 on dist-test-slave-47qt"
server_key: "e8fc1c7f52e108df26e91be82760e3df"
server_key_iv: "eb4c5d3dac0e6f689e6d9f6180c4cbc7"
server_key_version: "encryptionkey@0"
I20260618 23:24:32.906167 12258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance:
uuid: "ede25656177d49d694a7d6ba5f38b308"
format_stamp: "Formatted at 2026-06-18 23:24:32 on dist-test-slave-47qt"
server_key: "e8fc1c7f52e108df26e91be82760e3df"
server_key_iv: "eb4c5d3dac0e6f689e6d9f6180c4cbc7"
server_key_version: "encryptionkey@0"
I20260618 23:24:32.907480 12258 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:24:32.908269 12273 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:32.908514 12258 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:32.908617 12258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-2/wal
uuid: "ede25656177d49d694a7d6ba5f38b308"
format_stamp: "Formatted at 2026-06-18 23:24:32 on dist-test-slave-47qt"
server_key: "e8fc1c7f52e108df26e91be82760e3df"
server_key_iv: "eb4c5d3dac0e6f689e6d9f6180c4cbc7"
server_key_version: "encryptionkey@0"
I20260618 23:24:32.908699 12258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:32.917861 12258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:32.919589 12258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:32.919749 12258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:32.919962 12258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:32.920253 12258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:32.920305 12258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:32.920341 12258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:32.920367 12258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:32.926707 12258 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.131:37473
I20260618 23:24:32.926757 12386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.131:37473 every 8 connection(s)
I20260618 23:24:32.927044 12258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
Jun 18 23:24:32 dist-test-slave-47qt krb5kdc[11883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825072, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:32.936096 11967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:32.927755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:35531 (local address 127.8.39.190:39825)
0618 23:24:32.927872 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:32.927875 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:32.928064 (+ 189us) server_negotiation.cc:407] Connection header received
0618 23:24:32.928989 (+ 925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:32.928990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:32.929001 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:32.929056 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:32.929673 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:32.930167 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:32.931148 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:32.931337 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:32.931431 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:32.933480 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:32.933482 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:32.933482 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:32.933489 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:32.934766 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:32.935178 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:32.935179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:32.935180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:32.935236 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:32.935479 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:32.935480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:32.935481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:32.935588 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:32.935663 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:32.935929 (+ 266us) server_negotiation.cc:299] Negotiation successful
0618 23:24:32.936000 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260618 23:24:32.936308 12389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:32.927713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:39825 (local address 127.8.39.131:35531)
0618 23:24:32.927939 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:32.927982 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:32.928881 (+ 899us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:32.929140 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:32.929143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:32.929406 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:32.929583 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:32.929587 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:32.930267 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:32.930268 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:32.931050 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:32.931052 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:32.931454 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:32.931455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:32.931570 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:32.932011 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:32.932019 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:32.933375 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:32.934863 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:32.934876 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:32.934878 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:32.935090 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:32.935302 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:32.935303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:32.935303 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:32.935410 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:32.935655 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:32.935660 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:32.935789 (+ 129us) client_negotiation.cc:769] Sending connection context
0618 23:24:32.935907 (+ 118us) client_negotiation.cc:240] Negotiation successful
0618 23:24:32.936064 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":85,"threads_started":1}
I20260618 23:24:32.936343 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 12258
I20260618 23:24:32.936446 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance
I20260618 23:24:32.936643 8350 external_mini_cluster.cc:1428] Setting key c2d6365578cb22f50cc331c20d4ac9f5
I20260618 23:24:32.936779 12387 heartbeater.cc:347] Connected to a master server at 127.8.39.190:39825
I20260618 23:24:32.936882 12387 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:32.937067 12387 heartbeater.cc:511] Master 127.8.39.190:39825 requested a full tablet report, sending...
I20260618 23:24:32.937501 11927 ts_manager.cc:194] Registered new tserver with Master: ede25656177d49d694a7d6ba5f38b308 (127.8.39.131:37473)
I20260618 23:24:32.937891 11927 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.131@KRBTEST.COM'} at 127.8.39.131:35531
I20260618 23:24:32.948416 8350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 18 23:24:32 dist-test-slave-47qt krb5kdc[11883](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 18 23:24:32 dist-test-slave-47qt krb5kdc[11883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.129@KRBTEST.COM
I20260618 23:24:32.971674 12394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:32.964060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35466 (local address 127.8.39.129:42395)
0618 23:24:32.964251 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:32.964256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:32.964266 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:24:32.964291 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:32.964293 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:32.964307 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:32.964374 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:32.964911 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:32.965736 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:32.966477 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:32.966799 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:32.966862 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:32.968994 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:32.969002 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:32.969002 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:32.969009 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:32.970511 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:32.970847 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:32.970850 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:32.970850 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:32.970903 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:32.971174 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:32.971175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:32.971175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:32.971319 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:32.971436 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:32.971520 (+ 84us) server_negotiation.cc:299] Negotiation successful
0618 23:24:32.971595 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":90,"threads_started":1}
Jun 18 23:24:32 dist-test-slave-47qt krb5kdc[11883](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:32.980363 11967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:32.973374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32866 (local address 127.8.39.190:39825)
0618 23:24:32.973420 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:32.973422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:32.973455 (+ 33us) server_negotiation.cc:407] Connection header received
0618 23:24:32.973521 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:32.973522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:32.973530 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:32.973575 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:32.974286 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:32.974773 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:32.975419 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:32.975604 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:32.975670 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:32.977961 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:32.977962 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:32.977963 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:32.977976 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:32.979138 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:32.979517 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:32.979518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:32.979519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:32.979577 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:32.979768 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:32.979769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:32.979770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:32.979907 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:32.980019 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:32.980137 (+ 118us) server_negotiation.cc:299] Negotiation successful
0618 23:24:32.980247 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260618 23:24:32.980664 11927 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32866
I20260618 23:24:32.981163 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 11990
I20260618 23:24:32.985653 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 12124
I20260618 23:24:32.990289 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 12258
I20260618 23:24:32.994614 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 11896
2026-06-18T23:24:33Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1865 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:24:33 dist-test-slave-47qt krb5kdc[12399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:24:33 dist-test-slave-47qt krb5kdc[12399](info): set up 2 sockets
Jun 18 23:24:33 dist-test-slave-47qt krb5kdc[12399](info): commencing operation
krb5kdc: starting...
W20260618 23:24:34.049448 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:24:34 dist-test-slave-47qt krb5kdc[12399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:24:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:24:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.39.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
I20260618 23:24:34.168355 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:34395
--webserver_interface=127.8.39.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:35709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:34395
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:34.249837 12412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:34.250099 12412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:34.250137 12412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:34.251279 12412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:24:34.251350 12412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:34.251374 12412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:34.251401 12412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:24:34.251420 12412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:24:34.252882 12412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:35709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:34395
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:34395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:34.253185 12412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:34.253405 12412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:34.258139 12417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:34.258132 12420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:34.258131 12418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:34.258553 12412 server_base.cc:1034] running on GCE node
I20260618 23:24:34.258821 12412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:34.259105 12412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:34.260303 12412 hybrid_clock.cc:648] HybridClock initialized: now 1781825074260286 us; error 51 us; skew 500 ppm
Jun 18 23:24:34 dist-test-slave-47qt krb5kdc[12399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825074, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:34.265810 12412 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:24:34.266177 12412 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:34.266248 12412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:34.266387 12412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:34.267432 12412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "2fa5878935554e7ca7e4c07eb8e74d88"
format_stamp: "Formatted at 2026-06-18 23:24:34 on dist-test-slave-47qt"
server_key: "55b147eac2d39e344d8fc7a04ef9ce98"
server_key_iv: "3ce6e520d6a164de5c319b4f43001082"
server_key_version: "encryptionkey@0"
I20260618 23:24:34.267781 12412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "2fa5878935554e7ca7e4c07eb8e74d88"
format_stamp: "Formatted at 2026-06-18 23:24:34 on dist-test-slave-47qt"
server_key: "55b147eac2d39e344d8fc7a04ef9ce98"
server_key_iv: "3ce6e520d6a164de5c319b4f43001082"
server_key_version: "encryptionkey@0"
I20260618 23:24:34.269073 12412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260618 23:24:34.269852 12426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:34.270056 12412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:24:34.270156 12412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "2fa5878935554e7ca7e4c07eb8e74d88"
format_stamp: "Formatted at 2026-06-18 23:24:34 on dist-test-slave-47qt"
server_key: "55b147eac2d39e344d8fc7a04ef9ce98"
server_key_iv: "3ce6e520d6a164de5c319b4f43001082"
server_key_version: "encryptionkey@0"
I20260618 23:24:34.270215 12412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:34.283459 12412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:34.285847 12412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:34.285969 12412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:34.289880 12412 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:34395
I20260618 23:24:34.290091 12478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:34395 every 8 connection(s)
I20260618 23:24:34.290232 12412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:24:34.291155 12479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:34.293349 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 12412
I20260618 23:24:34.293476 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:24:34.293660 8350 external_mini_cluster.cc:1428] Setting key 7f9b6dc0e8f9b41e67a5ed8a64d3e4b2
I20260618 23:24:34.293779 12479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fa5878935554e7ca7e4c07eb8e74d88: Bootstrap starting.
I20260618 23:24:34.294520 12479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2fa5878935554e7ca7e4c07eb8e74d88: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:34.294852 12479 log.cc:824] T 00000000000000000000000000000000 P 2fa5878935554e7ca7e4c07eb8e74d88: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:34.295570 12479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2fa5878935554e7ca7e4c07eb8e74d88: No bootstrap required, opened a new log
I20260618 23:24:34.297917 12479 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2fa5878935554e7ca7e4c07eb8e74d88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa5878935554e7ca7e4c07eb8e74d88" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 34395 } }
I20260618 23:24:34.298077 12479 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2fa5878935554e7ca7e4c07eb8e74d88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:34.298106 12479 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2fa5878935554e7ca7e4c07eb8e74d88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fa5878935554e7ca7e4c07eb8e74d88, State: Initialized, Role: FOLLOWER
I20260618 23:24:34.298213 12479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2fa5878935554e7ca7e4c07eb8e74d88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa5878935554e7ca7e4c07eb8e74d88" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 34395 } }
I20260618 23:24:34.298281 12479 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2fa5878935554e7ca7e4c07eb8e74d88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:24:34.298309 12479 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2fa5878935554e7ca7e4c07eb8e74d88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:24:34.298357 12479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2fa5878935554e7ca7e4c07eb8e74d88 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:34.298992 12479 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2fa5878935554e7ca7e4c07eb8e74d88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa5878935554e7ca7e4c07eb8e74d88" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 34395 } }
I20260618 23:24:34.299115 12479 leader_election.cc:302] T 00000000000000000000000000000000 P 2fa5878935554e7ca7e4c07eb8e74d88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2fa5878935554e7ca7e4c07eb8e74d88; no voters:
I20260618 23:24:34.299333 12479 leader_election.cc:288] T 00000000000000000000000000000000 P 2fa5878935554e7ca7e4c07eb8e74d88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:24:34.299415 12484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2fa5878935554e7ca7e4c07eb8e74d88 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:34.299594 12479 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2fa5878935554e7ca7e4c07eb8e74d88 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:24:34.299607 12484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2fa5878935554e7ca7e4c07eb8e74d88 [term 1 LEADER]: Becoming Leader. State: Replica: 2fa5878935554e7ca7e4c07eb8e74d88, State: Running, Role: LEADER
I20260618 23:24:34.299813 12484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2fa5878935554e7ca7e4c07eb8e74d88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa5878935554e7ca7e4c07eb8e74d88" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 34395 } }
I20260618 23:24:34.300122 12484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fa5878935554e7ca7e4c07eb8e74d88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2fa5878935554e7ca7e4c07eb8e74d88. Latest consensus state: current_term: 1 leader_uuid: "2fa5878935554e7ca7e4c07eb8e74d88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa5878935554e7ca7e4c07eb8e74d88" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 34395 } } }
I20260618 23:24:34.300196 12484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fa5878935554e7ca7e4c07eb8e74d88 [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:34.300341 12486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2fa5878935554e7ca7e4c07eb8e74d88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2fa5878935554e7ca7e4c07eb8e74d88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2fa5878935554e7ca7e4c07eb8e74d88" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 34395 } } }
I20260618 23:24:34.300439 12486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2fa5878935554e7ca7e4c07eb8e74d88 [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:34.300554 12491 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 18 23:24:34 dist-test-slave-47qt krb5kdc[12399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825074, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:34.301435 12491 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:24:34.303114 12491 catalog_manager.cc:1347] Generated new cluster ID: 2a73cc95e4d14e1d8b0a5f8989b9f9f0
I20260618 23:24:34.303166 12491 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:24:34.305438 12482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:34.294206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35216 (local address 127.8.39.190:34395)
0618 23:24:34.294371 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:34.294379 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:34.294411 (+ 32us) server_negotiation.cc:407] Connection header received
0618 23:24:34.295015 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:34.295023 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:34.295096 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:34.295206 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:34.295847 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.297209 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.297917 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.298526 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.298636 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:34.301668 (+ 3032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:34.301672 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:34.301673 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:34.301685 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:34.303687 (+ 2002us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:34.304177 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:34.304181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:34.304181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:34.304286 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:34.304519 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:34.304520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:34.304520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:34.304819 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:34.304981 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:34.305060 (+ 79us) server_negotiation.cc:299] Negotiation successful
0618 23:24:34.305221 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":82,"threads_started":1}
I20260618 23:24:34.320371 12491 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:24:34.321164 12491 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:24:34.329327 12491 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2fa5878935554e7ca7e4c07eb8e74d88: Generated new TSK 0
I20260618 23:24:34.329556 12491 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.39.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
I20260618 23:24:34.361159 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.129:0
--local_ip_for_outbound_sockets=127.8.39.129
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:34395
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:35709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:34.442401 12505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:34.442625 12505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:34.442663 12505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:34.443799 12505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:34.443871 12505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:34.443917 12505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.129
I20260618 23:24:34.445453 12505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:35709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:34395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12505
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.129
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:34.445724 12505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:34.445996 12505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:34.450995 12513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:34.451002 12511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:34.451269 12505 server_base.cc:1034] running on GCE node
W20260618 23:24:34.451339 12510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:34.451546 12505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:34.451797 12505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:34.452970 12505 hybrid_clock.cc:648] HybridClock initialized: now 1781825074452958 us; error 29 us; skew 500 ppm
Jun 18 23:24:34 dist-test-slave-47qt krb5kdc[12399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825074, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:34.455668 12505 init.cc:377] Logged in from keytab as kudu/127.8.39.129@KRBTEST.COM (short username kudu)
I20260618 23:24:34.456192 12505 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:34.456271 12505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:34.456419 12505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:34.457568 12505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-0/data/instance:
uuid: "ce01570418a64356821f9d7e851d1122"
format_stamp: "Formatted at 2026-06-18 23:24:34 on dist-test-slave-47qt"
server_key: "8262a6d83f08545646424a5d2a2b92cb"
server_key_iv: "9263dd59beea271820b0a16b08af50c9"
server_key_version: "encryptionkey@0"
I20260618 23:24:34.457926 12505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance:
uuid: "ce01570418a64356821f9d7e851d1122"
format_stamp: "Formatted at 2026-06-18 23:24:34 on dist-test-slave-47qt"
server_key: "8262a6d83f08545646424a5d2a2b92cb"
server_key_iv: "9263dd59beea271820b0a16b08af50c9"
server_key_version: "encryptionkey@0"
I20260618 23:24:34.459297 12505 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:34.460177 12519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:34.460381 12505 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:34.460486 12505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-0/wal
uuid: "ce01570418a64356821f9d7e851d1122"
format_stamp: "Formatted at 2026-06-18 23:24:34 on dist-test-slave-47qt"
server_key: "8262a6d83f08545646424a5d2a2b92cb"
server_key_iv: "9263dd59beea271820b0a16b08af50c9"
server_key_version: "encryptionkey@0"
I20260618 23:24:34.460583 12505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:34.473812 12505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:34.476074 12505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:34.476230 12505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:34.476476 12505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:34.476774 12505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:34.476827 12505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:34.476867 12505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:34.476895 12505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:34.482743 12505 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.129:40019
I20260618 23:24:34.482797 12632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.129:40019 every 8 connection(s)
I20260618 23:24:34.483054 12505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
I20260618 23:24:34.486661 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 12505
I20260618 23:24:34.486753 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance
I20260618 23:24:34.487035 8350 external_mini_cluster.cc:1428] Setting key a8488cf215227e7c6c6860770001b8e1
Jun 18 23:24:34 dist-test-slave-47qt krb5kdc[12399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825074, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.39.130@KRBTEST.COM; defaulting to no policy
I20260618 23:24:34.495254 12482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:34.484473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:48429 (local address 127.8.39.190:34395)
0618 23:24:34.484563 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:34.484565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:34.484762 (+ 197us) server_negotiation.cc:407] Connection header received
0618 23:24:34.485480 (+ 718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:34.485482 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:34.485494 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:34.485554 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:34.486933 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.487565 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.489039 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.489300 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.489393 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:34.491864 (+ 2471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:34.491866 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:34.491866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:34.491876 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:34.493617 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:34.494100 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:34.494101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:34.494102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:34.494169 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:34.494487 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:34.494488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:34.494488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:34.494618 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:34.494705 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:34.495057 (+ 352us) server_negotiation.cc:299] Negotiation successful
0618 23:24:34.495152 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260618 23:24:34.495488 12635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:34.484459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:34395 (local address 127.8.39.129:48429)
0618 23:24:34.484620 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:34.484667 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:34.485365 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:34.485631 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:34.485634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:34.486226 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:34.486807 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:34.486814 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.487658 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.487659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:34.488946 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:34.488949 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.489410 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.489411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:34.489539 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:34.490103 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:34.490112 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:34.491742 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:34.493738 (+ 1996us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:34.493746 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:34.493749 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:34.494008 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:34.494247 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:34.494248 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:34.494249 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:34.494381 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:34.494694 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:34.494701 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:34.494891 (+ 190us) client_negotiation.cc:769] Sending connection context
0618 23:24:34.495037 (+ 146us) client_negotiation.cc:240] Negotiation successful
0618 23:24:34.495215 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260618 23:24:34.496130 12633 heartbeater.cc:347] Connected to a master server at 127.8.39.190:34395
I20260618 23:24:34.496227 12633 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:34.496351 12633 heartbeater.cc:511] Master 127.8.39.190:34395 requested a full tablet report, sending...
I20260618 23:24:34.496819 12441 ts_manager.cc:194] Registered new tserver with Master: ce01570418a64356821f9d7e851d1122 (127.8.39.129:40019)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.130@KRBTEST.COM" created.
I20260618 23:24:34.497674 12441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.129@KRBTEST.COM'} at 127.8.39.129:48429
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
I20260618 23:24:34.512907 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.130:0
--local_ip_for_outbound_sockets=127.8.39.130
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:34395
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:35709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:34.603483 12638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:34.603711 12638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:34.603739 12638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:34.604976 12638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:34.605014 12638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:34.605044 12638 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.130
I20260618 23:24:34.606721 12638 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:35709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:34395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.130
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:34.606926 12638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:34.607108 12638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:34.612571 12644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:34.612630 12643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:34.612567 12646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:34.612939 12638 server_base.cc:1034] running on GCE node
I20260618 23:24:34.613189 12638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:34.613440 12638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:34.614598 12638 hybrid_clock.cc:648] HybridClock initialized: now 1781825074614594 us; error 40 us; skew 500 ppm
Jun 18 23:24:34 dist-test-slave-47qt krb5kdc[12399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825074, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:34.617093 12638 init.cc:377] Logged in from keytab as kudu/127.8.39.130@KRBTEST.COM (short username kudu)
I20260618 23:24:34.617465 12638 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:34.617540 12638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:34.617910 12638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:34.619446 12638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-1/data/instance:
uuid: "6b2f80c3d4aa41aab1abf5c4e651f58d"
format_stamp: "Formatted at 2026-06-18 23:24:34 on dist-test-slave-47qt"
server_key: "e96edb6dbddfd90a7152dc0e852f5a2c"
server_key_iv: "92961001bcadd76f1669d8a1ab4bc6b4"
server_key_version: "encryptionkey@0"
I20260618 23:24:34.620146 12638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance:
uuid: "6b2f80c3d4aa41aab1abf5c4e651f58d"
format_stamp: "Formatted at 2026-06-18 23:24:34 on dist-test-slave-47qt"
server_key: "e96edb6dbddfd90a7152dc0e852f5a2c"
server_key_iv: "92961001bcadd76f1669d8a1ab4bc6b4"
server_key_version: "encryptionkey@0"
I20260618 23:24:34.621805 12638 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.004s sys 0.000s
I20260618 23:24:34.622577 12652 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:34.622802 12638 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:34.622915 12638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-1/wal
uuid: "6b2f80c3d4aa41aab1abf5c4e651f58d"
format_stamp: "Formatted at 2026-06-18 23:24:34 on dist-test-slave-47qt"
server_key: "e96edb6dbddfd90a7152dc0e852f5a2c"
server_key_iv: "92961001bcadd76f1669d8a1ab4bc6b4"
server_key_version: "encryptionkey@0"
I20260618 23:24:34.623003 12638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:34.635212 12638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:34.637382 12638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:34.637562 12638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:34.637820 12638 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:34.638139 12638 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:34.638181 12638 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:34.638238 12638 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:34.638276 12638 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:34.644881 12638 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.130:40105
I20260618 23:24:34.644909 12765 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.130:40105 every 8 connection(s)
I20260618 23:24:34.645200 12638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
I20260618 23:24:34.648376 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 12638
I20260618 23:24:34.648591 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance
I20260618 23:24:34.648823 8350 external_mini_cluster.cc:1428] Setting key c344f14797f5f3205b78f624af057006
Jun 18 23:24:34 dist-test-slave-47qt krb5kdc[12399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825074, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:34.655781 12482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:34.645913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:43021 (local address 127.8.39.190:34395)
0618 23:24:34.645978 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:34.645980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:34.646208 (+ 228us) server_negotiation.cc:407] Connection header received
0618 23:24:34.646669 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:34.646670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:34.646679 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:34.646731 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:34.647655 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.648245 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.649779 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.649993 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.650079 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:34.652753 (+ 2674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:34.652755 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:34.652755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:34.652762 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:34.654332 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:34.654817 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:34.654818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:34.654818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:34.654872 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:34.655119 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:34.655120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:34.655120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:34.655234 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:34.655310 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:34.655591 (+ 281us) server_negotiation.cc:299] Negotiation successful
0618 23:24:34.655670 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260618 23:24:34.655973 12768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:34.645898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:34395 (local address 127.8.39.130:43021)
0618 23:24:34.646086 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:34.646122 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:34.646588 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:34.646792 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:34.646794 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:34.647044 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:34.647554 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:34.647559 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.648346 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.648347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:34.649687 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:34.649689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.650089 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.650090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:34.650215 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:34.650967 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:34.650975 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:34.652616 (+ 1641us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:34.654469 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:34.654476 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:34.654478 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:34.654721 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:34.654942 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:34.654943 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:34.654943 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:34.655049 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:34.655302 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:34.655308 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:34.655459 (+ 151us) client_negotiation.cc:769] Sending connection context
0618 23:24:34.655570 (+ 111us) client_negotiation.cc:240] Negotiation successful
0618 23:24:34.655730 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":101,"threads_started":1}
I20260618 23:24:34.656461 12766 heartbeater.cc:347] Connected to a master server at 127.8.39.190:34395
I20260618 23:24:34.656549 12766 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:34.656702 12766 heartbeater.cc:511] Master 127.8.39.190:34395 requested a full tablet report, sending...
I20260618 23:24:34.657105 12441 ts_manager.cc:194] Registered new tserver with Master: 6b2f80c3d4aa41aab1abf5c4e651f58d (127.8.39.130:40105)
I20260618 23:24:34.657506 12441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.130@KRBTEST.COM'} at 127.8.39.130:43021
WARNING: no policy specified for kudu/127.8.39.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
I20260618 23:24:34.676646 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.131:0
--local_ip_for_outbound_sockets=127.8.39.131
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:34395
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:35709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:34.761193 12771 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:34.761407 12771 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:34.761431 12771 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:34.762627 12771 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:34.762672 12771 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:34.762702 12771 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.131
I20260618 23:24:34.764317 12771 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:35709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:34395
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12771
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.131
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:34.764511 12771 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:34.764684 12771 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:34.769723 12777 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:34.769686 12779 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:34.769767 12776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:34.769915 12771 server_base.cc:1034] running on GCE node
I20260618 23:24:34.770108 12771 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:34.770346 12771 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:34.771487 12771 hybrid_clock.cc:648] HybridClock initialized: now 1781825074771476 us; error 29 us; skew 500 ppm
Jun 18 23:24:34 dist-test-slave-47qt krb5kdc[12399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825074, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:34.774235 12771 init.cc:377] Logged in from keytab as kudu/127.8.39.131@KRBTEST.COM (short username kudu)
I20260618 23:24:34.774612 12771 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:34.774662 12771 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:34.774838 12771 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:34.775897 12771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-2/data/instance:
uuid: "5bbbeb1c9dcf439ebc77cf6a68e2bec5"
format_stamp: "Formatted at 2026-06-18 23:24:34 on dist-test-slave-47qt"
server_key: "ae2c22b8cec486ed13b8272dc249848c"
server_key_iv: "2c71e74f4d6aeddb874ef21bdfc2a449"
server_key_version: "encryptionkey@0"
I20260618 23:24:34.776252 12771 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance:
uuid: "5bbbeb1c9dcf439ebc77cf6a68e2bec5"
format_stamp: "Formatted at 2026-06-18 23:24:34 on dist-test-slave-47qt"
server_key: "ae2c22b8cec486ed13b8272dc249848c"
server_key_iv: "2c71e74f4d6aeddb874ef21bdfc2a449"
server_key_version: "encryptionkey@0"
I20260618 23:24:34.777783 12771 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:24:34.778712 12785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:34.778932 12771 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:34.779047 12771 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-2/wal
uuid: "5bbbeb1c9dcf439ebc77cf6a68e2bec5"
format_stamp: "Formatted at 2026-06-18 23:24:34 on dist-test-slave-47qt"
server_key: "ae2c22b8cec486ed13b8272dc249848c"
server_key_iv: "2c71e74f4d6aeddb874ef21bdfc2a449"
server_key_version: "encryptionkey@0"
I20260618 23:24:34.779129 12771 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:34.802649 12771 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:34.804927 12771 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:34.805081 12771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:34.805346 12771 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:34.806061 12771 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:34.806136 12771 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:34.806183 12771 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:34.806210 12771 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:34.813571 12771 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.131:45169
I20260618 23:24:34.813647 12898 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.131:45169 every 8 connection(s)
I20260618 23:24:34.813918 12771 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
Jun 18 23:24:34 dist-test-slave-47qt krb5kdc[12399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825074, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:34.822598 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 12771
I20260618 23:24:34.822731 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance
I20260618 23:24:34.822913 8350 external_mini_cluster.cc:1428] Setting key 84060892e4eeacc739920d07e863aea6
I20260618 23:24:34.823386 12482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:34.814668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:38489 (local address 127.8.39.190:34395)
0618 23:24:34.814763 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:34.814765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:34.814891 (+ 126us) server_negotiation.cc:407] Connection header received
0618 23:24:34.815414 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:34.815415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:34.815424 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:34.815491 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:34.816467 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.816968 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.818068 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.818261 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.818328 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:34.820418 (+ 2090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:34.820420 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:34.820420 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:34.820426 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:34.821783 (+ 1357us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:34.822198 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:34.822199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:34.822199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:34.822246 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:34.822536 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:34.822536 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:34.822536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:34.822639 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:34.822727 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:34.823203 (+ 476us) server_negotiation.cc:299] Negotiation successful
0618 23:24:34.823299 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260618 23:24:34.823578 12901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:34.814604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:34395 (local address 127.8.39.131:38489)
0618 23:24:34.814770 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:34.814810 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:34.815319 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:34.815545 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:34.815549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:34.815817 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:34.816314 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:34.816319 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.817061 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.817083 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:34.817981 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:34.817983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.818361 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.818362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:34.818469 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:34.818956 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:34.818964 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:34.820331 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:34.821878 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:34.821884 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:34.821886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:34.822122 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:34.822307 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:34.822308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:34.822308 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:34.822459 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:34.822720 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:34.822727 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:34.822914 (+ 187us) client_negotiation.cc:769] Sending connection context
0618 23:24:34.823159 (+ 245us) client_negotiation.cc:240] Negotiation successful
0618 23:24:34.823332 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260618 23:24:34.824066 12899 heartbeater.cc:347] Connected to a master server at 127.8.39.190:34395
I20260618 23:24:34.824153 12899 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:34.824306 12899 heartbeater.cc:511] Master 127.8.39.190:34395 requested a full tablet report, sending...
I20260618 23:24:34.824769 12441 ts_manager.cc:194] Registered new tserver with Master: 5bbbeb1c9dcf439ebc77cf6a68e2bec5 (127.8.39.131:45169)
I20260618 23:24:34.825191 12441 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.131@KRBTEST.COM'} at 127.8.39.131:38489
I20260618 23:24:34.835304 8350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:24:34.843559 12482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:34.836924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35232 (local address 127.8.39.190:34395)
0618 23:24:34.837095 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:34.837097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:34.837106 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:24:34.837174 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:34.837175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:34.837185 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:34.837250 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:34.837802 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.838344 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.839235 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.839488 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.839595 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:34.840953 (+ 1358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:34.840954 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:34.840954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:34.840963 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:34.842391 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:34.842729 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:34.842729 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:34.842730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:34.842787 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:34.843015 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:34.843016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:34.843016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:34.843173 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:34.843268 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:34.843417 (+ 149us) server_negotiation.cc:299] Negotiation successful
0618 23:24:34.843489 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260618 23:24:34.845110 12441 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35232:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:24:34.846767 12441 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:24:34.854163 12911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:34.849104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:40105 (local address 127.0.0.1:40898)
0618 23:24:34.849637 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:34.849647 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:34.849759 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:34.850561 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:34.850561 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:34.850563 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:34.850621 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:34.850623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.851782 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.851783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:34.852899 (+ 1116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:34.852902 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.853938 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.853939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:34.854011 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:34.854015 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:24:34.854055 (+ 40us) client_negotiation.cc:240] Negotiation successful
0618 23:24:34.854099 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"mutex_wait_us":35,"thread_start_us":132,"threads_started":1}
I20260618 23:24:34.854252 12913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:34.849393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46542 (local address 127.8.39.129:40019)
0618 23:24:34.849577 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:34.849582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:34.849594 (+ 12us) server_negotiation.cc:407] Connection header received
0618 23:24:34.849651 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:34.849654 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:34.849672 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:34.849784 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:34.850108 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.851377 (+ 1269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.853237 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.853769 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.853882 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:34.853948 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:34.854120 (+ 172us) server_negotiation.cc:299] Negotiation successful
0618 23:24:34.854167 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":78,"threads_started":1}
I20260618 23:24:34.854313 12910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:34.849104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:40019 (local address 127.0.0.1:46542)
0618 23:24:34.849273 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:34.849317 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:34.849399 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:34.849875 (+ 476us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:34.849885 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:34.849890 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:34.849986 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:34.849989 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.851505 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.851506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:34.852896 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:34.852898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.853886 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.853888 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:34.853975 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:34.853989 (+ 14us) client_negotiation.cc:769] Sending connection context
0618 23:24:34.854102 (+ 113us) client_negotiation.cc:240] Negotiation successful
0618 23:24:34.854149 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
I20260618 23:24:34.854465 12912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:34.849380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:45169 (local address 127.0.0.1:55490)
0618 23:24:34.850092 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:34.850110 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:34.850293 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:34.850513 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:34.850514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:34.850516 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:34.850560 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:34.850562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.851875 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.851876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:34.852991 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:34.852993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.854199 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.854200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:34.854264 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:34.854269 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:24:34.854328 (+ 59us) client_negotiation.cc:240] Negotiation successful
0618 23:24:34.854394 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":72,"threads_started":1}
I20260618 23:24:34.854856 12914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:34.849398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40898 (local address 127.8.39.130:40105)
0618 23:24:34.850419 (+ 1021us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:34.850424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:34.850432 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:24:34.850469 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:34.850471 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:34.850487 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:34.850563 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:34.850679 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.851669 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.852999 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.853823 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.854430 (+ 607us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:34.854657 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:34.854689 (+ 32us) server_negotiation.cc:299] Negotiation successful
0618 23:24:34.854745 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":987,"thread_start_us":161,"threads_started":1}
I20260618 23:24:34.855003 12915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:34.849594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55490 (local address 127.8.39.131:45169)
0618 23:24:34.849803 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:34.849807 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:34.850307 (+ 500us) server_negotiation.cc:407] Connection header received
0618 23:24:34.850377 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:34.850379 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:34.850397 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:34.850629 (+ 232us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:34.850705 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.851772 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.853060 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.853823 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.853912 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:34.853974 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:34.854873 (+ 899us) server_negotiation.cc:299] Negotiation successful
0618 23:24:34.854922 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":89,"threads_started":1}
I20260618 23:24:34.855355 12567 tablet_service.cc:1467] Processing CreateTablet for tablet 50a411e8c53b44969adf35e86b90a26c (DEFAULT_TABLE table=test-table [id=700876b4b07b4a058356b46e52841f32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:34.855639 12567 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50a411e8c53b44969adf35e86b90a26c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:34.855674 12833 tablet_service.cc:1467] Processing CreateTablet for tablet 50a411e8c53b44969adf35e86b90a26c (DEFAULT_TABLE table=test-table [id=700876b4b07b4a058356b46e52841f32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:34.855942 12833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50a411e8c53b44969adf35e86b90a26c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:34.857946 12700 tablet_service.cc:1467] Processing CreateTablet for tablet 50a411e8c53b44969adf35e86b90a26c (DEFAULT_TABLE table=test-table [id=700876b4b07b4a058356b46e52841f32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:34.858281 12700 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 50a411e8c53b44969adf35e86b90a26c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:34.858675 12916 tablet_bootstrap.cc:492] T 50a411e8c53b44969adf35e86b90a26c P ce01570418a64356821f9d7e851d1122: Bootstrap starting.
I20260618 23:24:34.858973 12917 tablet_bootstrap.cc:492] T 50a411e8c53b44969adf35e86b90a26c P 5bbbeb1c9dcf439ebc77cf6a68e2bec5: Bootstrap starting.
I20260618 23:24:34.859469 12916 tablet_bootstrap.cc:654] T 50a411e8c53b44969adf35e86b90a26c P ce01570418a64356821f9d7e851d1122: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:34.859653 12917 tablet_bootstrap.cc:654] T 50a411e8c53b44969adf35e86b90a26c P 5bbbeb1c9dcf439ebc77cf6a68e2bec5: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:34.860065 12916 log.cc:824] T 50a411e8c53b44969adf35e86b90a26c P ce01570418a64356821f9d7e851d1122: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:34.860288 12917 log.cc:824] T 50a411e8c53b44969adf35e86b90a26c P 5bbbeb1c9dcf439ebc77cf6a68e2bec5: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:34.860891 12916 tablet_bootstrap.cc:492] T 50a411e8c53b44969adf35e86b90a26c P ce01570418a64356821f9d7e851d1122: No bootstrap required, opened a new log
I20260618 23:24:34.860970 12916 ts_tablet_manager.cc:1397] T 50a411e8c53b44969adf35e86b90a26c P ce01570418a64356821f9d7e851d1122: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260618 23:24:34.861073 12917 tablet_bootstrap.cc:492] T 50a411e8c53b44969adf35e86b90a26c P 5bbbeb1c9dcf439ebc77cf6a68e2bec5: No bootstrap required, opened a new log
I20260618 23:24:34.861174 12917 ts_tablet_manager.cc:1397] T 50a411e8c53b44969adf35e86b90a26c P 5bbbeb1c9dcf439ebc77cf6a68e2bec5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260618 23:24:34.861894 12920 tablet_bootstrap.cc:492] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d: Bootstrap starting.
I20260618 23:24:34.862613 12920 tablet_bootstrap.cc:654] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:34.863099 12916 raft_consensus.cc:348] T 50a411e8c53b44969adf35e86b90a26c P ce01570418a64356821f9d7e851d1122 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce01570418a64356821f9d7e851d1122" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40019 } } peers { permanent_uuid: "6b2f80c3d4aa41aab1abf5c4e651f58d" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 40105 } } peers { permanent_uuid: "5bbbeb1c9dcf439ebc77cf6a68e2bec5" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 45169 } }
I20260618 23:24:34.863292 12920 log.cc:824] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:34.863323 12916 raft_consensus.cc:374] T 50a411e8c53b44969adf35e86b90a26c P ce01570418a64356821f9d7e851d1122 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:34.863369 12916 raft_consensus.cc:729] T 50a411e8c53b44969adf35e86b90a26c P ce01570418a64356821f9d7e851d1122 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce01570418a64356821f9d7e851d1122, State: Initialized, Role: FOLLOWER
I20260618 23:24:34.863320 12917 raft_consensus.cc:348] T 50a411e8c53b44969adf35e86b90a26c P 5bbbeb1c9dcf439ebc77cf6a68e2bec5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce01570418a64356821f9d7e851d1122" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40019 } } peers { permanent_uuid: "6b2f80c3d4aa41aab1abf5c4e651f58d" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 40105 } } peers { permanent_uuid: "5bbbeb1c9dcf439ebc77cf6a68e2bec5" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 45169 } }
I20260618 23:24:34.863487 12917 raft_consensus.cc:374] T 50a411e8c53b44969adf35e86b90a26c P 5bbbeb1c9dcf439ebc77cf6a68e2bec5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:34.863541 12917 raft_consensus.cc:729] T 50a411e8c53b44969adf35e86b90a26c P 5bbbeb1c9dcf439ebc77cf6a68e2bec5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bbbeb1c9dcf439ebc77cf6a68e2bec5, State: Initialized, Role: FOLLOWER
I20260618 23:24:34.863513 12916 consensus_queue.cc:260] T 50a411e8c53b44969adf35e86b90a26c P ce01570418a64356821f9d7e851d1122 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce01570418a64356821f9d7e851d1122" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40019 } } peers { permanent_uuid: "6b2f80c3d4aa41aab1abf5c4e651f58d" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 40105 } } peers { permanent_uuid: "5bbbeb1c9dcf439ebc77cf6a68e2bec5" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 45169 } }
I20260618 23:24:34.863680 12917 consensus_queue.cc:260] T 50a411e8c53b44969adf35e86b90a26c P 5bbbeb1c9dcf439ebc77cf6a68e2bec5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce01570418a64356821f9d7e851d1122" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40019 } } peers { permanent_uuid: "6b2f80c3d4aa41aab1abf5c4e651f58d" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 40105 } } peers { permanent_uuid: "5bbbeb1c9dcf439ebc77cf6a68e2bec5" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 45169 } }
I20260618 23:24:34.863867 12633 heartbeater.cc:503] Master 127.8.39.190:34395 was elected leader, sending a full tablet report...
I20260618 23:24:34.863994 12917 ts_tablet_manager.cc:1428] T 50a411e8c53b44969adf35e86b90a26c P 5bbbeb1c9dcf439ebc77cf6a68e2bec5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260618 23:24:34.864005 12899 heartbeater.cc:503] Master 127.8.39.190:34395 was elected leader, sending a full tablet report...
I20260618 23:24:34.864104 12916 ts_tablet_manager.cc:1428] T 50a411e8c53b44969adf35e86b90a26c P ce01570418a64356821f9d7e851d1122: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:24:34.864811 12920 tablet_bootstrap.cc:492] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d: No bootstrap required, opened a new log
I20260618 23:24:34.864900 12920 ts_tablet_manager.cc:1397] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 23:24:34.866580 12920 raft_consensus.cc:348] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce01570418a64356821f9d7e851d1122" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40019 } } peers { permanent_uuid: "6b2f80c3d4aa41aab1abf5c4e651f58d" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 40105 } } peers { permanent_uuid: "5bbbeb1c9dcf439ebc77cf6a68e2bec5" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 45169 } }
I20260618 23:24:34.866709 12920 raft_consensus.cc:374] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:34.866732 12920 raft_consensus.cc:729] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b2f80c3d4aa41aab1abf5c4e651f58d, State: Initialized, Role: FOLLOWER
I20260618 23:24:34.866856 12920 consensus_queue.cc:260] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce01570418a64356821f9d7e851d1122" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40019 } } peers { permanent_uuid: "6b2f80c3d4aa41aab1abf5c4e651f58d" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 40105 } } peers { permanent_uuid: "5bbbeb1c9dcf439ebc77cf6a68e2bec5" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 45169 } }
I20260618 23:24:34.867148 12920 ts_tablet_manager.cc:1428] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260618 23:24:34.867146 12766 heartbeater.cc:503] Master 127.8.39.190:34395 was elected leader, sending a full tablet report...
W20260618 23:24:34.895778 12767 tablet.cc:2369] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:24:34.979413 12924 raft_consensus.cc:482] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:24:34.979591 12924 raft_consensus.cc:504] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce01570418a64356821f9d7e851d1122" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40019 } } peers { permanent_uuid: "6b2f80c3d4aa41aab1abf5c4e651f58d" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 40105 } } peers { permanent_uuid: "5bbbeb1c9dcf439ebc77cf6a68e2bec5" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 45169 } }
I20260618 23:24:34.980106 12924 leader_election.cc:288] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce01570418a64356821f9d7e851d1122 (127.8.39.129:40019), 5bbbeb1c9dcf439ebc77cf6a68e2bec5 (127.8.39.131:45169)
W20260618 23:24:34.983738 12634 tablet.cc:2369] T 50a411e8c53b44969adf35e86b90a26c P ce01570418a64356821f9d7e851d1122: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:24:34.984086 12915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:34.980604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:40525 (local address 127.8.39.131:45169)
0618 23:24:34.980787 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:34.980789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:34.980799 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:24:34.980829 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:34.980830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:34.980845 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:34.980917 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:34.981089 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.981944 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.983267 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.983713 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.983787 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:34.983797 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:34.983986 (+ 189us) server_negotiation.cc:299] Negotiation successful
0618 23:24:34.984035 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260618 23:24:34.984335 12925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:34.980462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:45169 (local address 127.8.39.130:40525)
0618 23:24:34.980687 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:34.980698 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:34.980769 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:34.980939 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:34.980940 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:34.980942 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:34.981010 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:34.981012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.982053 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.982054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:34.983172 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:34.983174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.983832 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.983833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:34.983902 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:34.983906 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:24:34.984201 (+ 295us) client_negotiation.cc:240] Negotiation successful
0618 23:24:34.984248 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":132,"threads_started":1}
I20260618 23:24:34.984426 12768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:34.980352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:40019 (local address 127.8.39.130:38727)
0618 23:24:34.980441 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:34.980455 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:34.980510 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:34.980848 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:34.980849 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:34.980851 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:34.980927 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:34.980930 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.981729 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.981730 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:34.982940 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:34.982943 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.983966 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.983967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:34.984082 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:34.984086 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:24:34.984153 (+ 67us) client_negotiation.cc:240] Negotiation successful
0618 23:24:34.984204 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260618 23:24:34.984452 12913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:34.980615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:38727 (local address 127.8.39.129:40019)
0618 23:24:34.980683 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:34.980685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:34.980696 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:24:34.980741 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:34.980745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:34.980760 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:34.980818 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:34.981010 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.981625 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.983025 (+ 1400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:34.983710 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:34.983787 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:34.983797 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:34.984341 (+ 544us) server_negotiation.cc:299] Negotiation successful
0618 23:24:34.984388 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260618 23:24:34.984822 12587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a411e8c53b44969adf35e86b90a26c" candidate_uuid: "6b2f80c3d4aa41aab1abf5c4e651f58d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce01570418a64356821f9d7e851d1122" is_pre_election: true
I20260618 23:24:34.984845 12853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a411e8c53b44969adf35e86b90a26c" candidate_uuid: "6b2f80c3d4aa41aab1abf5c4e651f58d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bbbeb1c9dcf439ebc77cf6a68e2bec5" is_pre_election: true
I20260618 23:24:34.984994 12853 raft_consensus.cc:2454] T 50a411e8c53b44969adf35e86b90a26c P 5bbbeb1c9dcf439ebc77cf6a68e2bec5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b2f80c3d4aa41aab1abf5c4e651f58d in term 0.
I20260618 23:24:34.984997 12587 raft_consensus.cc:2454] T 50a411e8c53b44969adf35e86b90a26c P ce01570418a64356821f9d7e851d1122 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b2f80c3d4aa41aab1abf5c4e651f58d in term 0.
I20260618 23:24:34.985250 12654 leader_election.cc:302] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bbbeb1c9dcf439ebc77cf6a68e2bec5, 6b2f80c3d4aa41aab1abf5c4e651f58d; no voters:
I20260618 23:24:34.985397 12924 raft_consensus.cc:2781] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:24:34.985455 12924 raft_consensus.cc:482] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:24:34.985486 12924 raft_consensus.cc:3037] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:34.986160 12924 raft_consensus.cc:504] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce01570418a64356821f9d7e851d1122" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40019 } } peers { permanent_uuid: "6b2f80c3d4aa41aab1abf5c4e651f58d" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 40105 } } peers { permanent_uuid: "5bbbeb1c9dcf439ebc77cf6a68e2bec5" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 45169 } }
I20260618 23:24:34.986321 12924 leader_election.cc:288] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d [CANDIDATE]: Term 1 election: Requested vote from peers ce01570418a64356821f9d7e851d1122 (127.8.39.129:40019), 5bbbeb1c9dcf439ebc77cf6a68e2bec5 (127.8.39.131:45169)
I20260618 23:24:34.986467 12853 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a411e8c53b44969adf35e86b90a26c" candidate_uuid: "6b2f80c3d4aa41aab1abf5c4e651f58d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5bbbeb1c9dcf439ebc77cf6a68e2bec5"
I20260618 23:24:34.986500 12587 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "50a411e8c53b44969adf35e86b90a26c" candidate_uuid: "6b2f80c3d4aa41aab1abf5c4e651f58d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce01570418a64356821f9d7e851d1122"
I20260618 23:24:34.986572 12853 raft_consensus.cc:3037] T 50a411e8c53b44969adf35e86b90a26c P 5bbbeb1c9dcf439ebc77cf6a68e2bec5 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:34.986584 12587 raft_consensus.cc:3037] T 50a411e8c53b44969adf35e86b90a26c P ce01570418a64356821f9d7e851d1122 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:34.987253 12853 raft_consensus.cc:2454] T 50a411e8c53b44969adf35e86b90a26c P 5bbbeb1c9dcf439ebc77cf6a68e2bec5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b2f80c3d4aa41aab1abf5c4e651f58d in term 1.
I20260618 23:24:34.987412 12654 leader_election.cc:302] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5bbbeb1c9dcf439ebc77cf6a68e2bec5, 6b2f80c3d4aa41aab1abf5c4e651f58d; no voters:
I20260618 23:24:34.987537 12924 raft_consensus.cc:2781] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:34.987627 12587 raft_consensus.cc:2454] T 50a411e8c53b44969adf35e86b90a26c P ce01570418a64356821f9d7e851d1122 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6b2f80c3d4aa41aab1abf5c4e651f58d in term 1.
I20260618 23:24:34.987731 12924 raft_consensus.cc:686] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d [term 1 LEADER]: Becoming Leader. State: Replica: 6b2f80c3d4aa41aab1abf5c4e651f58d, State: Running, Role: LEADER
I20260618 23:24:34.987829 12924 consensus_queue.cc:237] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce01570418a64356821f9d7e851d1122" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40019 } } peers { permanent_uuid: "6b2f80c3d4aa41aab1abf5c4e651f58d" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 40105 } } peers { permanent_uuid: "5bbbeb1c9dcf439ebc77cf6a68e2bec5" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 45169 } }
I20260618 23:24:34.988660 12441 catalog_manager.cc:5495] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6b2f80c3d4aa41aab1abf5c4e651f58d (127.8.39.130). New cstate: current_term: 1 leader_uuid: "6b2f80c3d4aa41aab1abf5c4e651f58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce01570418a64356821f9d7e851d1122" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6b2f80c3d4aa41aab1abf5c4e651f58d" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 40105 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5bbbeb1c9dcf439ebc77cf6a68e2bec5" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 45169 } health_report { overall_health: UNKNOWN } } }
I20260618 23:24:35.004717 12914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:35.002399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40904 (local address 127.8.39.130:40105)
0618 23:24:35.002442 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:35.002444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:35.002459 (+ 15us) server_negotiation.cc:407] Connection header received
0618 23:24:35.002490 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:35.002491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:35.002507 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:35.002573 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:24:35.002701 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:35.003197 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:35.003967 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:35.004169 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:35.004236 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:35.004404 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:24:35.004503 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:24:35.004580 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:35.004642 (+ 62us) server_negotiation.cc:299] Negotiation successful
0618 23:24:35.004661 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260618 23:24:35.006673 12587 raft_consensus.cc:1261] T 50a411e8c53b44969adf35e86b90a26c P ce01570418a64356821f9d7e851d1122 [term 1 FOLLOWER]: Refusing update from remote peer 6b2f80c3d4aa41aab1abf5c4e651f58d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:24:35.006677 12853 raft_consensus.cc:1261] T 50a411e8c53b44969adf35e86b90a26c P 5bbbeb1c9dcf439ebc77cf6a68e2bec5 [term 1 FOLLOWER]: Refusing update from remote peer 6b2f80c3d4aa41aab1abf5c4e651f58d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:24:35.007050 12926 consensus_queue.cc:1035] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce01570418a64356821f9d7e851d1122" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:35.007171 12924 consensus_queue.cc:1035] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "5bbbeb1c9dcf439ebc77cf6a68e2bec5" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 45169 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:35.009704 12930 mvcc.cc:205] Tried to move back new op lower bound from 7298355507224416256 to 7298355507150839808. Current Snapshot: MvccSnapshot[applied={T|T < 7298355507224416256}]
I20260618 23:24:35.012611 12441 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:24:35.012732 12441 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35232:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:24:35.013633 12441 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2fa5878935554e7ca7e4c07eb8e74d88: Sending DeleteTablet for 3 replicas of tablet 50a411e8c53b44969adf35e86b90a26c
I20260618 23:24:35.014024 12700 tablet_service.cc:1514] Processing DeleteTablet for tablet 50a411e8c53b44969adf35e86b90a26c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:35 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:40898
I20260618 23:24:35.014021 12833 tablet_service.cc:1514] Processing DeleteTablet for tablet 50a411e8c53b44969adf35e86b90a26c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:35 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:55490
I20260618 23:24:35.014249 12938 tablet_replica.cc:330] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d: stopping tablet replica
I20260618 23:24:35.014266 12567 tablet_service.cc:1514] Processing DeleteTablet for tablet 50a411e8c53b44969adf35e86b90a26c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:35 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:46542
I20260618 23:24:35.014340 12938 raft_consensus.cc:2229] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:24:35.014394 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 12505
I20260618 23:24:35.014474 12938 raft_consensus.cc:2258] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:35.014626 12939 tablet_replica.cc:330] T 50a411e8c53b44969adf35e86b90a26c P 5bbbeb1c9dcf439ebc77cf6a68e2bec5: stopping tablet replica
I20260618 23:24:35.014712 12939 raft_consensus.cc:2229] T 50a411e8c53b44969adf35e86b90a26c P 5bbbeb1c9dcf439ebc77cf6a68e2bec5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:24:35.014823 12939 raft_consensus.cc:2258] T 50a411e8c53b44969adf35e86b90a26c P 5bbbeb1c9dcf439ebc77cf6a68e2bec5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:35.015350 12939 ts_tablet_manager.cc:1905] T 50a411e8c53b44969adf35e86b90a26c P 5bbbeb1c9dcf439ebc77cf6a68e2bec5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:35.017045 12939 ts_tablet_manager.cc:1918] T 50a411e8c53b44969adf35e86b90a26c P 5bbbeb1c9dcf439ebc77cf6a68e2bec5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:24:35.017150 12939 log.cc:1196] T 50a411e8c53b44969adf35e86b90a26c P 5bbbeb1c9dcf439ebc77cf6a68e2bec5: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-2/wal/wals/50a411e8c53b44969adf35e86b90a26c
I20260618 23:24:35.017499 12939 ts_tablet_manager.cc:1939] T 50a411e8c53b44969adf35e86b90a26c P 5bbbeb1c9dcf439ebc77cf6a68e2bec5: Deleting consensus metadata
I20260618 23:24:35.017954 12428 catalog_manager.cc:4841] TS 5bbbeb1c9dcf439ebc77cf6a68e2bec5 (127.8.39.131:45169): tablet 50a411e8c53b44969adf35e86b90a26c (table test-table [id=700876b4b07b4a058356b46e52841f32]) successfully deleted
I20260618 23:24:35.018096 12938 ts_tablet_manager.cc:1905] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:35.019626 12938 ts_tablet_manager.cc:1918] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:24:35.019703 12938 log.cc:1196] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781825053542692-8350-0/minicluster-data/ts-1/wal/wals/50a411e8c53b44969adf35e86b90a26c
I20260618 23:24:35.019992 12938 ts_tablet_manager.cc:1939] T 50a411e8c53b44969adf35e86b90a26c P 6b2f80c3d4aa41aab1abf5c4e651f58d: Deleting consensus metadata
I20260618 23:24:35.020390 12428 catalog_manager.cc:4841] TS 6b2f80c3d4aa41aab1abf5c4e651f58d (127.8.39.130:40105): tablet 50a411e8c53b44969adf35e86b90a26c (table test-table [id=700876b4b07b4a058356b46e52841f32]) successfully deleted
W20260618 23:24:35.021673 12430 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.39.129:40019 (error 108)
I20260618 23:24:35.022010 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 12638
I20260618 23:24:35.022177 12912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:35.022058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:40019 (local address 127.0.0.1:46548)
0618 23:24:35.022108 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:35.022142 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.129:40019: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260618 23:24:35.023433 12430 catalog_manager.cc:4584] TS ce01570418a64356821f9d7e851d1122 (127.8.39.129:40019): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 50a411e8c53b44969adf35e86b90a26c: Network error: Client connection negotiation failed: client connection to 127.8.39.129:40019: connect: Connection refused (error 111)
I20260618 23:24:35.026990 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 12771
I20260618 23:24:35.031234 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 12412
2026-06-18T23:24:35Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2036 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-18T23:24:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:24:35Z Disabled control of system clock
I20260618 23:24:35.059849 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:36833
--webserver_interface=127.8.39.190
--webserver_port=0
--builtin_ntp_servers=127.8.39.148:35867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:36833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260618 23:24:35.148126 12945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:35.148324 12945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:35.148344 12945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:35.149427 12945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:24:35.149463 12945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:35.149477 12945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:35.149489 12945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:24:35.149501 12945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:24:35.150760 12945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:35867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:36833
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:36833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:35.150952 12945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:35.151180 12945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:35.156068 12953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:35.156068 12951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:35.156222 12950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:35.156342 12945 server_base.cc:1034] running on GCE node
I20260618 23:24:35.156555 12945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:35.156847 12945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:35.158041 12945 hybrid_clock.cc:648] HybridClock initialized: now 1781825075157990 us; error 77 us; skew 500 ppm
I20260618 23:24:35.159487 12945 webserver.cc:460] Webserver started at http://127.8.39.190:34451/ using document root <none> and password file <none>
I20260618 23:24:35.159759 12945 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:35.159837 12945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:35.159992 12945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:35.161085 12945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "29958908d79f4e99981793a688a77cbc"
format_stamp: "Formatted at 2026-06-18 23:24:35 on dist-test-slave-47qt"
server_key: "6e4beb35f9a4834efee8467e7c7307c2"
server_key_iv: "57432e891092b5f1489a6eedb48b5678"
server_key_version: "encryptionkey@0"
I20260618 23:24:35.161518 12945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "29958908d79f4e99981793a688a77cbc"
format_stamp: "Formatted at 2026-06-18 23:24:35 on dist-test-slave-47qt"
server_key: "6e4beb35f9a4834efee8467e7c7307c2"
server_key_iv: "57432e891092b5f1489a6eedb48b5678"
server_key_version: "encryptionkey@0"
I20260618 23:24:35.162912 12945 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:35.163815 12959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:35.164124 12945 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:35.164229 12945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "29958908d79f4e99981793a688a77cbc"
format_stamp: "Formatted at 2026-06-18 23:24:35 on dist-test-slave-47qt"
server_key: "6e4beb35f9a4834efee8467e7c7307c2"
server_key_iv: "57432e891092b5f1489a6eedb48b5678"
server_key_version: "encryptionkey@0"
I20260618 23:24:35.164305 12945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:35.167584 12945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:35.167831 12945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:35.167971 12945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:35.171712 12945 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:36833
I20260618 23:24:35.171763 13011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:36833 every 8 connection(s)
I20260618 23:24:35.172086 12945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:24:35.172744 13012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:35.174247 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 12945
I20260618 23:24:35.174374 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:24:35.174563 8350 external_mini_cluster.cc:1428] Setting key 4461c11fd38ea964d4c26c5456592de8
I20260618 23:24:35.175933 13012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29958908d79f4e99981793a688a77cbc: Bootstrap starting.
I20260618 23:24:35.176494 13012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29958908d79f4e99981793a688a77cbc: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:35.176795 13012 log.cc:824] T 00000000000000000000000000000000 P 29958908d79f4e99981793a688a77cbc: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:35.177603 13012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29958908d79f4e99981793a688a77cbc: No bootstrap required, opened a new log
I20260618 23:24:35.179350 13015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:35.175314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39578 (local address 127.8.39.190:36833)
0618 23:24:35.175536 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:35.175546 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:35.177209 (+ 1663us) server_negotiation.cc:407] Connection header received
0618 23:24:35.177824 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:35.177832 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:35.177879 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:35.178035 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:35.178758 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:35.178761 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:35.178762 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:24:35.178766 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:35.178812 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:24:35.178825 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:35.178918 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:35.179024 (+ 106us) server_negotiation.cc:299] Negotiation successful
0618 23:24:35.179080 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":82,"threads_started":1}
I20260618 23:24:35.179551 13012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29958908d79f4e99981793a688a77cbc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29958908d79f4e99981793a688a77cbc" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 36833 } }
I20260618 23:24:35.179854 13012 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29958908d79f4e99981793a688a77cbc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:35.179900 13012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29958908d79f4e99981793a688a77cbc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29958908d79f4e99981793a688a77cbc, State: Initialized, Role: FOLLOWER
I20260618 23:24:35.180019 13012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29958908d79f4e99981793a688a77cbc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29958908d79f4e99981793a688a77cbc" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 36833 } }
I20260618 23:24:35.180121 13012 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29958908d79f4e99981793a688a77cbc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:24:35.180178 13012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29958908d79f4e99981793a688a77cbc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:24:35.180228 13012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29958908d79f4e99981793a688a77cbc [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:35.181079 13012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29958908d79f4e99981793a688a77cbc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29958908d79f4e99981793a688a77cbc" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 36833 } }
I20260618 23:24:35.181247 13012 leader_election.cc:302] T 00000000000000000000000000000000 P 29958908d79f4e99981793a688a77cbc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29958908d79f4e99981793a688a77cbc; no voters:
I20260618 23:24:35.181429 13012 leader_election.cc:288] T 00000000000000000000000000000000 P 29958908d79f4e99981793a688a77cbc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:24:35.181492 13017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29958908d79f4e99981793a688a77cbc [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:35.181690 13012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29958908d79f4e99981793a688a77cbc [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:24:35.181689 13017 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29958908d79f4e99981793a688a77cbc [term 1 LEADER]: Becoming Leader. State: Replica: 29958908d79f4e99981793a688a77cbc, State: Running, Role: LEADER
I20260618 23:24:35.181859 13017 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29958908d79f4e99981793a688a77cbc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29958908d79f4e99981793a688a77cbc" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 36833 } }
I20260618 23:24:35.182227 13018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29958908d79f4e99981793a688a77cbc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29958908d79f4e99981793a688a77cbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29958908d79f4e99981793a688a77cbc" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 36833 } } }
I20260618 23:24:35.182236 13019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29958908d79f4e99981793a688a77cbc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29958908d79f4e99981793a688a77cbc. Latest consensus state: current_term: 1 leader_uuid: "29958908d79f4e99981793a688a77cbc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29958908d79f4e99981793a688a77cbc" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 36833 } } }
I20260618 23:24:35.182363 13018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29958908d79f4e99981793a688a77cbc [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:35.182406 13019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29958908d79f4e99981793a688a77cbc [sys.catalog]: This master's current role is: LEADER
W20260618 23:24:35.183449 13033 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 29958908d79f4e99981793a688a77cbc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 23:24:35.183506 13033 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:24:35.183677 13026 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:24:35.183868 13026 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:24:35.185227 13026 catalog_manager.cc:1347] Generated new cluster ID: 537d35a14992414c902427ef11ce2d6e
I20260618 23:24:35.185299 13026 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:24:35.189400 13026 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:24:35.189551 13026 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:24:35.195983 13026 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29958908d79f4e99981793a688a77cbc: Generated new TSK 0
I20260618 23:24:35.196161 13026 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:24:35.198680 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.129:0
--local_ip_for_outbound_sockets=127.8.39.129
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:36833
--builtin_ntp_servers=127.8.39.148:35867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260618 23:24:35.295423 13036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:35.295666 13036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:35.295703 13036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:35.296890 13036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:35.296958 13036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:35.297004 13036 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.129
I20260618 23:24:35.298401 13036 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:35867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:36833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13036
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.129
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:35.298676 13036 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:35.298987 13036 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:35.304064 13042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:35.304061 13041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:35.304064 13044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:35.304728 13036 server_base.cc:1034] running on GCE node
I20260618 23:24:35.304934 13036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:35.305238 13036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:35.306424 13036 hybrid_clock.cc:648] HybridClock initialized: now 1781825075306410 us; error 34 us; skew 500 ppm
I20260618 23:24:35.307920 13036 webserver.cc:460] Webserver started at http://127.8.39.129:41471/ using document root <none> and password file <none>
I20260618 23:24:35.308218 13036 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:35.308285 13036 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:35.308409 13036 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:35.309819 13036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-0/data/instance:
uuid: "37ee37589b6b43c5a3b2efe7c10b6a50"
format_stamp: "Formatted at 2026-06-18 23:24:35 on dist-test-slave-47qt"
server_key: "2d6e99d0678a0a0d5e693d79d438b214"
server_key_iv: "75b632251453ecc50c7211277841a1b5"
server_key_version: "encryptionkey@0"
I20260618 23:24:35.310230 13036 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance:
uuid: "37ee37589b6b43c5a3b2efe7c10b6a50"
format_stamp: "Formatted at 2026-06-18 23:24:35 on dist-test-slave-47qt"
server_key: "2d6e99d0678a0a0d5e693d79d438b214"
server_key_iv: "75b632251453ecc50c7211277841a1b5"
server_key_version: "encryptionkey@0"
I20260618 23:24:35.311985 13036 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260618 23:24:35.312947 13050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:35.313174 13036 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:35.313285 13036 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-0/wal
uuid: "37ee37589b6b43c5a3b2efe7c10b6a50"
format_stamp: "Formatted at 2026-06-18 23:24:35 on dist-test-slave-47qt"
server_key: "2d6e99d0678a0a0d5e693d79d438b214"
server_key_iv: "75b632251453ecc50c7211277841a1b5"
server_key_version: "encryptionkey@0"
I20260618 23:24:35.313364 13036 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:35.316582 13036 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:35.316862 13036 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:35.317003 13036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:35.317262 13036 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:35.317564 13036 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:35.317616 13036 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:35.317654 13036 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:35.317680 13036 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:35.323129 13036 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.129:38429
I20260618 23:24:35.323184 13163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.129:38429 every 8 connection(s)
I20260618 23:24:35.323539 13036 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
I20260618 23:24:35.323853 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 13036
I20260618 23:24:35.323961 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance
I20260618 23:24:35.324172 8350 external_mini_cluster.cc:1428] Setting key 0744b3fa4da0202774431753fe12983e
I20260618 23:24:35.325206 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.130:0
--local_ip_for_outbound_sockets=127.8.39.130
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:36833
--builtin_ntp_servers=127.8.39.148:35867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260618 23:24:35.327104 13015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:35.324408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:38869 (local address 127.8.39.190:36833)
0618 23:24:35.324452 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:35.324454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:35.324843 (+ 389us) server_negotiation.cc:407] Connection header received
0618 23:24:35.325491 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:35.325492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:35.325499 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:35.325543 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:35.326773 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:35.326774 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:35.326775 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:24:35.326781 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:35.326798 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:24:35.326803 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:35.326839 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:35.326997 (+ 158us) server_negotiation.cc:299] Negotiation successful
0618 23:24:35.327016 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260618 23:24:35.327273 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:35.324415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:36833 (local address 127.8.39.129:38869)
0618 23:24:35.324700 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:35.324756 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:35.325397 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:35.325609 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:35.325613 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:35.325622 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:35.326584 (+ 962us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:24:35.326584 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:35.326611 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:24:35.326637 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:35.326883 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:35.326888 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:35.326891 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:24:35.326968 (+ 77us) client_negotiation.cc:240] Negotiation successful
0618 23:24:35.327033 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":88,"threads_started":1}
I20260618 23:24:35.327868 13164 heartbeater.cc:347] Connected to a master server at 127.8.39.190:36833
I20260618 23:24:35.327970 13164 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:35.328130 13164 heartbeater.cc:511] Master 127.8.39.190:36833 requested a full tablet report, sending...
I20260618 23:24:35.328712 12976 ts_manager.cc:194] Registered new tserver with Master: 37ee37589b6b43c5a3b2efe7c10b6a50 (127.8.39.129:38429)
W20260618 23:24:35.406955 13167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:35.407194 13167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:35.407231 13167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:35.408361 13167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:35.408429 13167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:35.408475 13167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.130
I20260618 23:24:35.409900 13167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:35867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:36833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.130
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:35.410166 13167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:35.410504 13167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:35.416209 13172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:35.416293 13167 server_base.cc:1034] running on GCE node
W20260618 23:24:35.416280 13175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:35.416253 13173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:35.416620 13167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:35.416890 13167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:35.418079 13167 hybrid_clock.cc:648] HybridClock initialized: now 1781825075418055 us; error 114 us; skew 500 ppm
I20260618 23:24:35.419458 13167 webserver.cc:460] Webserver started at http://127.8.39.130:45171/ using document root <none> and password file <none>
I20260618 23:24:35.419710 13167 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:35.419790 13167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:35.419945 13167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:35.421032 13167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-1/data/instance:
uuid: "def016287d2a48c5b59d552164d1e243"
format_stamp: "Formatted at 2026-06-18 23:24:35 on dist-test-slave-47qt"
server_key: "8ffd09baa135f3271761a34896aba194"
server_key_iv: "b013a45322361782684d706c62f42e03"
server_key_version: "encryptionkey@0"
I20260618 23:24:35.421450 13167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance:
uuid: "def016287d2a48c5b59d552164d1e243"
format_stamp: "Formatted at 2026-06-18 23:24:35 on dist-test-slave-47qt"
server_key: "8ffd09baa135f3271761a34896aba194"
server_key_iv: "b013a45322361782684d706c62f42e03"
server_key_version: "encryptionkey@0"
I20260618 23:24:35.422748 13167 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:35.423593 13181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:35.423795 13167 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:35.423902 13167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-1/wal
uuid: "def016287d2a48c5b59d552164d1e243"
format_stamp: "Formatted at 2026-06-18 23:24:35 on dist-test-slave-47qt"
server_key: "8ffd09baa135f3271761a34896aba194"
server_key_iv: "b013a45322361782684d706c62f42e03"
server_key_version: "encryptionkey@0"
I20260618 23:24:35.423986 13167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:35.427888 13167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:35.428161 13167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:35.428310 13167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:35.428519 13167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:35.428836 13167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:35.428893 13167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:35.428930 13167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:35.428967 13167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:35.434819 13167 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.130:46371
I20260618 23:24:35.434861 13294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.130:46371 every 8 connection(s)
I20260618 23:24:35.435168 13167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
I20260618 23:24:35.438118 13015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:35.435851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:42147 (local address 127.8.39.190:36833)
0618 23:24:35.435911 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:35.435914 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:35.436123 (+ 209us) server_negotiation.cc:407] Connection header received
0618 23:24:35.436598 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:35.436599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:35.436605 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:35.436710 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:35.437908 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:35.437909 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:35.437909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:24:35.437913 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:35.437928 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:24:35.437931 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:35.437995 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:35.438054 (+ 59us) server_negotiation.cc:299] Negotiation successful
0618 23:24:35.438066 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260618 23:24:35.438289 13297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:35.435838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:36833 (local address 127.8.39.130:42147)
0618 23:24:35.436010 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:35.436050 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:35.436491 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:35.436736 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:35.436739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:35.436746 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:35.437742 (+ 996us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:24:35.437743 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:35.437769 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:24:35.437804 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:35.437995 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:35.437998 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:35.438000 (+ 2us) client_negotiation.cc:769] Sending connection context
0618 23:24:35.438051 (+ 51us) client_negotiation.cc:240] Negotiation successful
0618 23:24:35.438096 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260618 23:24:35.438833 13295 heartbeater.cc:347] Connected to a master server at 127.8.39.190:36833
I20260618 23:24:35.438984 13295 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:35.439167 13295 heartbeater.cc:511] Master 127.8.39.190:36833 requested a full tablet report, sending...
I20260618 23:24:35.439565 12976 ts_manager.cc:194] Registered new tserver with Master: def016287d2a48c5b59d552164d1e243 (127.8.39.130:46371)
I20260618 23:24:35.440948 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 13167
I20260618 23:24:35.441023 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance
I20260618 23:24:35.441253 8350 external_mini_cluster.cc:1428] Setting key a5d723908b1fd90d3d4b8962bc818bbe
I20260618 23:24:35.442108 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.131:0
--local_ip_for_outbound_sockets=127.8.39.131
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:36833
--builtin_ntp_servers=127.8.39.148:35867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260618 23:24:35.522647 13298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:35.522832 13298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:35.522853 13298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:35.523909 13298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:35.523941 13298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:35.523967 13298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.131
I20260618 23:24:35.525251 13298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:35867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:36833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13298
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.131
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:35.525413 13298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:35.525619 13298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:35.530444 13304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:35.530453 13303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:35.530463 13306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:35.530817 13298 server_base.cc:1034] running on GCE node
I20260618 23:24:35.531016 13298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:35.531250 13298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:35.532395 13298 hybrid_clock.cc:648] HybridClock initialized: now 1781825075532381 us; error 35 us; skew 500 ppm
I20260618 23:24:35.533615 13298 webserver.cc:460] Webserver started at http://127.8.39.131:34671/ using document root <none> and password file <none>
I20260618 23:24:35.533843 13298 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:35.533888 13298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:35.534013 13298 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:35.535048 13298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-2/data/instance:
uuid: "daec3f238f514f0e84a213fffb6ba49c"
format_stamp: "Formatted at 2026-06-18 23:24:35 on dist-test-slave-47qt"
server_key: "e9520af19d13457c71c0bdc13d24c342"
server_key_iv: "f6cfdb37602e1bd1c44acf22b0664dea"
server_key_version: "encryptionkey@0"
I20260618 23:24:35.535406 13298 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance:
uuid: "daec3f238f514f0e84a213fffb6ba49c"
format_stamp: "Formatted at 2026-06-18 23:24:35 on dist-test-slave-47qt"
server_key: "e9520af19d13457c71c0bdc13d24c342"
server_key_iv: "f6cfdb37602e1bd1c44acf22b0664dea"
server_key_version: "encryptionkey@0"
I20260618 23:24:35.536684 13298 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260618 23:24:35.537461 13312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:35.537670 13298 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:35.537768 13298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-2/wal
uuid: "daec3f238f514f0e84a213fffb6ba49c"
format_stamp: "Formatted at 2026-06-18 23:24:35 on dist-test-slave-47qt"
server_key: "e9520af19d13457c71c0bdc13d24c342"
server_key_iv: "f6cfdb37602e1bd1c44acf22b0664dea"
server_key_version: "encryptionkey@0"
I20260618 23:24:35.537848 13298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:35.541000 13298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:35.541288 13298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:35.541436 13298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:35.541647 13298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:35.541945 13298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:35.541999 13298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:35.542039 13298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:35.542068 13298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:35.547540 13298 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.131:41741
I20260618 23:24:35.547624 13425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.131:41741 every 8 connection(s)
I20260618 23:24:35.547921 13298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
I20260618 23:24:35.551047 13015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:35.548624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:35365 (local address 127.8.39.190:36833)
0618 23:24:35.548690 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:35.548693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:35.548958 (+ 265us) server_negotiation.cc:407] Connection header received
0618 23:24:35.549480 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:35.549481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:35.549486 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:35.549541 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:35.550744 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:35.550745 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:35.550745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:24:35.550749 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:35.550765 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:24:35.550769 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:35.550830 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:35.550976 (+ 146us) server_negotiation.cc:299] Negotiation successful
0618 23:24:35.550990 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260618 23:24:35.551118 13428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:35.548649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:36833 (local address 127.8.39.131:35365)
0618 23:24:35.548835 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:35.548866 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:35.549360 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:35.549604 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:35.549607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:35.549615 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:35.550554 (+ 939us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:24:35.550554 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:35.550579 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:24:35.550605 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:35.550851 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:35.550856 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:35.550859 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:24:35.550928 (+ 69us) client_negotiation.cc:240] Negotiation successful
0618 23:24:35.550974 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":84,"threads_started":1}
I20260618 23:24:35.551592 13426 heartbeater.cc:347] Connected to a master server at 127.8.39.190:36833
I20260618 23:24:35.551682 13426 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:35.551839 13426 heartbeater.cc:511] Master 127.8.39.190:36833 requested a full tablet report, sending...
I20260618 23:24:35.552157 12976 ts_manager.cc:194] Registered new tserver with Master: daec3f238f514f0e84a213fffb6ba49c (127.8.39.131:41741)
I20260618 23:24:35.556443 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 13298
I20260618 23:24:35.556532 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance
I20260618 23:24:35.556679 8350 external_mini_cluster.cc:1428] Setting key c37820dbb7396f565bea97eb170ee968
I20260618 23:24:35.557475 8350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:24:35.559624 13015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:35.558758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39592 (local address 127.8.39.190:36833)
0618 23:24:35.558817 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:35.558819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:35.558922 (+ 103us) server_negotiation.cc:407] Connection header received
0618 23:24:35.558961 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:35.558961 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:35.558965 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:35.559021 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:35.559465 (+ 444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:35.559466 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:35.559466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:24:35.559470 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:35.559480 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:24:35.559484 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:35.559528 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:35.559562 (+ 34us) server_negotiation.cc:299] Negotiation successful
0618 23:24:35.559572 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260618 23:24:35.561048 12976 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39592:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:24:35.563040 12976 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:24:35.567010 13435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:35.565383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:41741 (local address 127.0.0.1:44126)
0618 23:24:35.565548 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:35.565678 (+ 130us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:35.565754 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:35.566080 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:35.566081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:35.566083 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:35.566564 (+ 481us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:24:35.566565 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:35.566579 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:24:35.566603 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:35.566829 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:35.566834 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:35.566837 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:24:35.566928 (+ 91us) client_negotiation.cc:240] Negotiation successful
0618 23:24:35.566942 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":64,"threads_started":1}
I20260618 23:24:35.567024 13436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:35.565461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44126 (local address 127.8.39.131:41741)
0618 23:24:35.565952 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:35.565955 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:35.565962 (+ 7us) server_negotiation.cc:407] Connection header received
0618 23:24:35.565979 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:35.565981 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:35.565987 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:35.566070 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:35.566685 (+ 615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:35.566692 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:35.566692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:24:35.566697 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:35.566727 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:24:35.566747 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:35.566783 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:35.566938 (+ 155us) server_negotiation.cc:299] Negotiation successful
0618 23:24:35.566953 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":467,"thread_start_us":81,"threads_started":1}
I20260618 23:24:35.567536 13438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:35.565584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:46371 (local address 127.0.0.1:51452)
0618 23:24:35.565801 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:35.565811 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:35.565858 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:35.566639 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:35.566640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:35.566643 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:35.567059 (+ 416us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:24:35.567060 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:35.567067 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:24:35.567078 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:35.567407 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:35.567408 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:35.567409 (+ 1us) client_negotiation.cc:769] Sending connection context
0618 23:24:35.567468 (+ 59us) client_negotiation.cc:240] Negotiation successful
0618 23:24:35.567480 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":61,"threads_started":1}
I20260618 23:24:35.567709 13439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:35.565676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51452 (local address 127.8.39.130:46371)
0618 23:24:35.566491 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:35.566495 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:35.566504 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:24:35.566529 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:35.566531 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:35.566539 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:35.566619 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:35.567263 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:35.567269 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:35.567270 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:24:35.567274 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:35.567306 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:24:35.567325 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:35.567407 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:35.567486 (+ 79us) server_negotiation.cc:299] Negotiation successful
0618 23:24:35.567503 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":782,"thread_start_us":81,"threads_started":1}
I20260618 23:24:35.567895 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:35.565383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:38429 (local address 127.0.0.1:51646)
0618 23:24:35.565701 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:35.565710 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:35.565754 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:35.565967 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:35.565972 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:35.565978 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:35.566672 (+ 694us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:24:35.566673 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:35.566679 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:24:35.566692 (+ 13us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:35.567507 (+ 815us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:35.567507 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:35.567508 (+ 1us) client_negotiation.cc:769] Sending connection context
0618 23:24:35.567553 (+ 45us) client_negotiation.cc:240] Negotiation successful
0618 23:24:35.567565 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":2,"thread_start_us":76,"threads_started":1}
I20260618 23:24:35.567764 13360 tablet_service.cc:1467] Processing CreateTablet for tablet c23265c87f4c46339d5ba1a8453acea9 (DEFAULT_TABLE table=test-table [id=31e9b4542f384b1798a5209518c7f49e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:35.568126 13437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:35.565549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51646 (local address 127.8.39.129:38429)
0618 23:24:35.565730 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:35.565736 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:35.565748 (+ 12us) server_negotiation.cc:407] Connection header received
0618 23:24:35.565866 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:35.565873 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:35.565882 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:35.565966 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:35.567041 (+ 1075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:35.567049 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:35.567050 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:24:35.567365 (+ 315us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:35.567399 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:24:35.567426 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:35.567480 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:35.568011 (+ 531us) server_negotiation.cc:299] Negotiation successful
0618 23:24:35.568030 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":76,"threads_started":1}
I20260618 23:24:35.568153 13360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c23265c87f4c46339d5ba1a8453acea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:35.568295 13229 tablet_service.cc:1467] Processing CreateTablet for tablet c23265c87f4c46339d5ba1a8453acea9 (DEFAULT_TABLE table=test-table [id=31e9b4542f384b1798a5209518c7f49e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:35.568583 13229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c23265c87f4c46339d5ba1a8453acea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:35.568718 13098 tablet_service.cc:1467] Processing CreateTablet for tablet c23265c87f4c46339d5ba1a8453acea9 (DEFAULT_TABLE table=test-table [id=31e9b4542f384b1798a5209518c7f49e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:35.569017 13098 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c23265c87f4c46339d5ba1a8453acea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:35.571422 13440 tablet_bootstrap.cc:492] T c23265c87f4c46339d5ba1a8453acea9 P daec3f238f514f0e84a213fffb6ba49c: Bootstrap starting.
I20260618 23:24:35.571705 13442 tablet_bootstrap.cc:492] T c23265c87f4c46339d5ba1a8453acea9 P def016287d2a48c5b59d552164d1e243: Bootstrap starting.
I20260618 23:24:35.571698 13441 tablet_bootstrap.cc:492] T c23265c87f4c46339d5ba1a8453acea9 P 37ee37589b6b43c5a3b2efe7c10b6a50: Bootstrap starting.
I20260618 23:24:35.572139 13440 tablet_bootstrap.cc:654] T c23265c87f4c46339d5ba1a8453acea9 P daec3f238f514f0e84a213fffb6ba49c: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:35.572283 13442 tablet_bootstrap.cc:654] T c23265c87f4c46339d5ba1a8453acea9 P def016287d2a48c5b59d552164d1e243: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:35.572324 13441 tablet_bootstrap.cc:654] T c23265c87f4c46339d5ba1a8453acea9 P 37ee37589b6b43c5a3b2efe7c10b6a50: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:35.572463 13440 log.cc:824] T c23265c87f4c46339d5ba1a8453acea9 P daec3f238f514f0e84a213fffb6ba49c: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:35.572605 13441 log.cc:824] T c23265c87f4c46339d5ba1a8453acea9 P 37ee37589b6b43c5a3b2efe7c10b6a50: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:35.572614 13442 log.cc:824] T c23265c87f4c46339d5ba1a8453acea9 P def016287d2a48c5b59d552164d1e243: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:35.573307 13440 tablet_bootstrap.cc:492] T c23265c87f4c46339d5ba1a8453acea9 P daec3f238f514f0e84a213fffb6ba49c: No bootstrap required, opened a new log
I20260618 23:24:35.573376 13440 ts_tablet_manager.cc:1397] T c23265c87f4c46339d5ba1a8453acea9 P daec3f238f514f0e84a213fffb6ba49c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260618 23:24:35.573473 13441 tablet_bootstrap.cc:492] T c23265c87f4c46339d5ba1a8453acea9 P 37ee37589b6b43c5a3b2efe7c10b6a50: No bootstrap required, opened a new log
I20260618 23:24:35.573568 13441 ts_tablet_manager.cc:1397] T c23265c87f4c46339d5ba1a8453acea9 P 37ee37589b6b43c5a3b2efe7c10b6a50: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:24:35.573715 13442 tablet_bootstrap.cc:492] T c23265c87f4c46339d5ba1a8453acea9 P def016287d2a48c5b59d552164d1e243: No bootstrap required, opened a new log
I20260618 23:24:35.573779 13442 ts_tablet_manager.cc:1397] T c23265c87f4c46339d5ba1a8453acea9 P def016287d2a48c5b59d552164d1e243: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:24:35.575500 13441 raft_consensus.cc:348] T c23265c87f4c46339d5ba1a8453acea9 P 37ee37589b6b43c5a3b2efe7c10b6a50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daec3f238f514f0e84a213fffb6ba49c" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 41741 } } peers { permanent_uuid: "37ee37589b6b43c5a3b2efe7c10b6a50" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38429 } } peers { permanent_uuid: "def016287d2a48c5b59d552164d1e243" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46371 } }
I20260618 23:24:35.575652 13441 raft_consensus.cc:374] T c23265c87f4c46339d5ba1a8453acea9 P 37ee37589b6b43c5a3b2efe7c10b6a50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:35.575701 13441 raft_consensus.cc:729] T c23265c87f4c46339d5ba1a8453acea9 P 37ee37589b6b43c5a3b2efe7c10b6a50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37ee37589b6b43c5a3b2efe7c10b6a50, State: Initialized, Role: FOLLOWER
I20260618 23:24:35.575898 13441 consensus_queue.cc:260] T c23265c87f4c46339d5ba1a8453acea9 P 37ee37589b6b43c5a3b2efe7c10b6a50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daec3f238f514f0e84a213fffb6ba49c" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 41741 } } peers { permanent_uuid: "37ee37589b6b43c5a3b2efe7c10b6a50" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38429 } } peers { permanent_uuid: "def016287d2a48c5b59d552164d1e243" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46371 } }
I20260618 23:24:35.576079 13446 raft_consensus.cc:482] T c23265c87f4c46339d5ba1a8453acea9 P 37ee37589b6b43c5a3b2efe7c10b6a50 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:24:35.576133 13441 ts_tablet_manager.cc:1428] T c23265c87f4c46339d5ba1a8453acea9 P 37ee37589b6b43c5a3b2efe7c10b6a50: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:24:35.576056 13440 raft_consensus.cc:348] T c23265c87f4c46339d5ba1a8453acea9 P daec3f238f514f0e84a213fffb6ba49c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daec3f238f514f0e84a213fffb6ba49c" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 41741 } } peers { permanent_uuid: "37ee37589b6b43c5a3b2efe7c10b6a50" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38429 } } peers { permanent_uuid: "def016287d2a48c5b59d552164d1e243" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46371 } }
I20260618 23:24:35.576218 13440 raft_consensus.cc:374] T c23265c87f4c46339d5ba1a8453acea9 P daec3f238f514f0e84a213fffb6ba49c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:35.576242 13164 heartbeater.cc:503] Master 127.8.39.190:36833 was elected leader, sending a full tablet report...
I20260618 23:24:35.576265 13440 raft_consensus.cc:729] T c23265c87f4c46339d5ba1a8453acea9 P daec3f238f514f0e84a213fffb6ba49c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: daec3f238f514f0e84a213fffb6ba49c, State: Initialized, Role: FOLLOWER
I20260618 23:24:35.576335 13446 raft_consensus.cc:504] T c23265c87f4c46339d5ba1a8453acea9 P 37ee37589b6b43c5a3b2efe7c10b6a50 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daec3f238f514f0e84a213fffb6ba49c" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 41741 } } peers { permanent_uuid: "37ee37589b6b43c5a3b2efe7c10b6a50" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38429 } } peers { permanent_uuid: "def016287d2a48c5b59d552164d1e243" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46371 } }
I20260618 23:24:35.576331 13442 raft_consensus.cc:348] T c23265c87f4c46339d5ba1a8453acea9 P def016287d2a48c5b59d552164d1e243 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daec3f238f514f0e84a213fffb6ba49c" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 41741 } } peers { permanent_uuid: "37ee37589b6b43c5a3b2efe7c10b6a50" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38429 } } peers { permanent_uuid: "def016287d2a48c5b59d552164d1e243" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46371 } }
I20260618 23:24:35.576397 13440 consensus_queue.cc:260] T c23265c87f4c46339d5ba1a8453acea9 P daec3f238f514f0e84a213fffb6ba49c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daec3f238f514f0e84a213fffb6ba49c" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 41741 } } peers { permanent_uuid: "37ee37589b6b43c5a3b2efe7c10b6a50" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38429 } } peers { permanent_uuid: "def016287d2a48c5b59d552164d1e243" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46371 } }
I20260618 23:24:35.576467 13442 raft_consensus.cc:374] T c23265c87f4c46339d5ba1a8453acea9 P def016287d2a48c5b59d552164d1e243 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:35.576504 13442 raft_consensus.cc:729] T c23265c87f4c46339d5ba1a8453acea9 P def016287d2a48c5b59d552164d1e243 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: def016287d2a48c5b59d552164d1e243, State: Initialized, Role: FOLLOWER
I20260618 23:24:35.576623 13440 ts_tablet_manager.cc:1428] T c23265c87f4c46339d5ba1a8453acea9 P daec3f238f514f0e84a213fffb6ba49c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:24:35.576623 13442 consensus_queue.cc:260] T c23265c87f4c46339d5ba1a8453acea9 P def016287d2a48c5b59d552164d1e243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daec3f238f514f0e84a213fffb6ba49c" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 41741 } } peers { permanent_uuid: "37ee37589b6b43c5a3b2efe7c10b6a50" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38429 } } peers { permanent_uuid: "def016287d2a48c5b59d552164d1e243" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46371 } }
I20260618 23:24:35.576718 13426 heartbeater.cc:503] Master 127.8.39.190:36833 was elected leader, sending a full tablet report...
I20260618 23:24:35.576838 13442 ts_tablet_manager.cc:1428] T c23265c87f4c46339d5ba1a8453acea9 P def016287d2a48c5b59d552164d1e243: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:24:35.576951 13446 leader_election.cc:288] T c23265c87f4c46339d5ba1a8453acea9 P 37ee37589b6b43c5a3b2efe7c10b6a50 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers daec3f238f514f0e84a213fffb6ba49c (127.8.39.131:41741), def016287d2a48c5b59d552164d1e243 (127.8.39.130:46371)
I20260618 23:24:35.576988 13295 heartbeater.cc:503] Master 127.8.39.190:36833 was elected leader, sending a full tablet report...
I20260618 23:24:35.578302 13436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:35.577339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:35439 (local address 127.8.39.131:41741)
0618 23:24:35.577377 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:35.577379 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:35.577386 (+ 7us) server_negotiation.cc:407] Connection header received
0618 23:24:35.577404 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:35.577405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:35.577409 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:35.577462 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:35.578066 (+ 604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:35.578067 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:35.578067 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:24:35.578071 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:35.578086 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:24:35.578090 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:35.578141 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:35.578242 (+ 101us) server_negotiation.cc:299] Negotiation successful
0618 23:24:35.578261 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260618 23:24:35.578308 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:35.577077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:41741 (local address 127.8.39.129:35439)
0618 23:24:35.577209 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:35.577221 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:35.577256 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:35.577472 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:35.577473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:35.577476 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:35.577979 (+ 503us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:24:35.577979 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:35.577987 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:24:35.578002 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:35.578166 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:35.578167 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:35.578168 (+ 1us) client_negotiation.cc:769] Sending connection context
0618 23:24:35.578223 (+ 55us) client_negotiation.cc:240] Negotiation successful
0618 23:24:35.578235 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260618 23:24:35.578644 13380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c23265c87f4c46339d5ba1a8453acea9" candidate_uuid: "37ee37589b6b43c5a3b2efe7c10b6a50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daec3f238f514f0e84a213fffb6ba49c" is_pre_election: true
I20260618 23:24:35.578783 13380 raft_consensus.cc:2454] T c23265c87f4c46339d5ba1a8453acea9 P daec3f238f514f0e84a213fffb6ba49c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37ee37589b6b43c5a3b2efe7c10b6a50 in term 0.
I20260618 23:24:35.578935 13439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:35.577769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:57425 (local address 127.8.39.130:46371)
0618 23:24:35.577854 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:35.577856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:35.577996 (+ 140us) server_negotiation.cc:407] Connection header received
0618 23:24:35.578096 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:35.578097 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:35.578101 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:35.578131 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:35.578726 (+ 595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:35.578727 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:35.578727 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:24:35.578731 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:35.578744 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:24:35.578748 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:35.578777 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:35.578863 (+ 86us) server_negotiation.cc:299] Negotiation successful
0618 23:24:35.578878 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260618 23:24:35.578958 13450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:35.577560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:46371 (local address 127.8.39.129:57425)
0618 23:24:35.577940 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:35.577950 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:35.578014 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:35.578199 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:35.578200 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:35.578203 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:35.578615 (+ 412us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:24:35.578615 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:35.578636 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:24:35.578648 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:35.578814 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:35.578815 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:35.578815 (+ 0us) client_negotiation.cc:769] Sending connection context
0618 23:24:35.578855 (+ 40us) client_negotiation.cc:240] Negotiation successful
0618 23:24:35.578867 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":66,"threads_started":1}
I20260618 23:24:35.578996 13051 leader_election.cc:302] T c23265c87f4c46339d5ba1a8453acea9 P 37ee37589b6b43c5a3b2efe7c10b6a50 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37ee37589b6b43c5a3b2efe7c10b6a50, daec3f238f514f0e84a213fffb6ba49c; no voters:
I20260618 23:24:35.579121 13447 raft_consensus.cc:2781] T c23265c87f4c46339d5ba1a8453acea9 P 37ee37589b6b43c5a3b2efe7c10b6a50 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:24:35.579190 13447 raft_consensus.cc:482] T c23265c87f4c46339d5ba1a8453acea9 P 37ee37589b6b43c5a3b2efe7c10b6a50 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:24:35.579214 13447 raft_consensus.cc:3037] T c23265c87f4c46339d5ba1a8453acea9 P 37ee37589b6b43c5a3b2efe7c10b6a50 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:35.579327 13249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c23265c87f4c46339d5ba1a8453acea9" candidate_uuid: "37ee37589b6b43c5a3b2efe7c10b6a50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "def016287d2a48c5b59d552164d1e243" is_pre_election: true
I20260618 23:24:35.579437 13249 raft_consensus.cc:2454] T c23265c87f4c46339d5ba1a8453acea9 P def016287d2a48c5b59d552164d1e243 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 37ee37589b6b43c5a3b2efe7c10b6a50 in term 0.
I20260618 23:24:35.579954 13447 raft_consensus.cc:504] T c23265c87f4c46339d5ba1a8453acea9 P 37ee37589b6b43c5a3b2efe7c10b6a50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daec3f238f514f0e84a213fffb6ba49c" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 41741 } } peers { permanent_uuid: "37ee37589b6b43c5a3b2efe7c10b6a50" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38429 } } peers { permanent_uuid: "def016287d2a48c5b59d552164d1e243" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46371 } }
I20260618 23:24:35.580075 13447 leader_election.cc:288] T c23265c87f4c46339d5ba1a8453acea9 P 37ee37589b6b43c5a3b2efe7c10b6a50 [CANDIDATE]: Term 1 election: Requested vote from peers daec3f238f514f0e84a213fffb6ba49c (127.8.39.131:41741), def016287d2a48c5b59d552164d1e243 (127.8.39.130:46371)
I20260618 23:24:35.580235 13249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c23265c87f4c46339d5ba1a8453acea9" candidate_uuid: "37ee37589b6b43c5a3b2efe7c10b6a50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "def016287d2a48c5b59d552164d1e243"
I20260618 23:24:35.580322 13249 raft_consensus.cc:3037] T c23265c87f4c46339d5ba1a8453acea9 P def016287d2a48c5b59d552164d1e243 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:35.581159 13249 raft_consensus.cc:2454] T c23265c87f4c46339d5ba1a8453acea9 P def016287d2a48c5b59d552164d1e243 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37ee37589b6b43c5a3b2efe7c10b6a50 in term 1.
I20260618 23:24:35.581166 13380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c23265c87f4c46339d5ba1a8453acea9" candidate_uuid: "37ee37589b6b43c5a3b2efe7c10b6a50" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "daec3f238f514f0e84a213fffb6ba49c"
I20260618 23:24:35.581239 13380 raft_consensus.cc:3037] T c23265c87f4c46339d5ba1a8453acea9 P daec3f238f514f0e84a213fffb6ba49c [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:35.581322 13051 leader_election.cc:302] T c23265c87f4c46339d5ba1a8453acea9 P 37ee37589b6b43c5a3b2efe7c10b6a50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37ee37589b6b43c5a3b2efe7c10b6a50, def016287d2a48c5b59d552164d1e243; no voters:
I20260618 23:24:35.581444 13447 raft_consensus.cc:2781] T c23265c87f4c46339d5ba1a8453acea9 P 37ee37589b6b43c5a3b2efe7c10b6a50 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:35.581526 13447 raft_consensus.cc:686] T c23265c87f4c46339d5ba1a8453acea9 P 37ee37589b6b43c5a3b2efe7c10b6a50 [term 1 LEADER]: Becoming Leader. State: Replica: 37ee37589b6b43c5a3b2efe7c10b6a50, State: Running, Role: LEADER
I20260618 23:24:35.581632 13447 consensus_queue.cc:237] T c23265c87f4c46339d5ba1a8453acea9 P 37ee37589b6b43c5a3b2efe7c10b6a50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daec3f238f514f0e84a213fffb6ba49c" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 41741 } } peers { permanent_uuid: "37ee37589b6b43c5a3b2efe7c10b6a50" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38429 } } peers { permanent_uuid: "def016287d2a48c5b59d552164d1e243" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46371 } }
I20260618 23:24:35.582093 13380 raft_consensus.cc:2454] T c23265c87f4c46339d5ba1a8453acea9 P daec3f238f514f0e84a213fffb6ba49c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 37ee37589b6b43c5a3b2efe7c10b6a50 in term 1.
I20260618 23:24:35.582463 12976 catalog_manager.cc:5495] T c23265c87f4c46339d5ba1a8453acea9 P 37ee37589b6b43c5a3b2efe7c10b6a50 reported cstate change: term changed from 0 to 1, leader changed from <none> to 37ee37589b6b43c5a3b2efe7c10b6a50 (127.8.39.129). New cstate: current_term: 1 leader_uuid: "37ee37589b6b43c5a3b2efe7c10b6a50" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "daec3f238f514f0e84a213fffb6ba49c" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 41741 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "37ee37589b6b43c5a3b2efe7c10b6a50" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "def016287d2a48c5b59d552164d1e243" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46371 } health_report { overall_health: UNKNOWN } } }
I20260618 23:24:35.588541 13437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:35.587682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51662 (local address 127.8.39.129:38429)
0618 23:24:35.587753 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:35.587754 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:35.587764 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:24:35.587791 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:35.587791 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:35.587800 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:35.587856 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:35.588320 (+ 464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:35.588321 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:35.588321 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:24:35.588325 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:35.588336 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:24:35.588340 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:35.588384 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:35.588431 (+ 47us) server_negotiation.cc:299] Negotiation successful
0618 23:24:35.588441 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260618 23:24:35.590068 13249 raft_consensus.cc:1261] T c23265c87f4c46339d5ba1a8453acea9 P def016287d2a48c5b59d552164d1e243 [term 1 FOLLOWER]: Refusing update from remote peer 37ee37589b6b43c5a3b2efe7c10b6a50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:24:35.590082 13380 raft_consensus.cc:1261] T c23265c87f4c46339d5ba1a8453acea9 P daec3f238f514f0e84a213fffb6ba49c [term 1 FOLLOWER]: Refusing update from remote peer 37ee37589b6b43c5a3b2efe7c10b6a50: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:24:35.590380 13446 consensus_queue.cc:1035] T c23265c87f4c46339d5ba1a8453acea9 P 37ee37589b6b43c5a3b2efe7c10b6a50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "daec3f238f514f0e84a213fffb6ba49c" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 41741 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:35.590489 13447 consensus_queue.cc:1035] T c23265c87f4c46339d5ba1a8453acea9 P 37ee37589b6b43c5a3b2efe7c10b6a50 [LEADER]: Connected to new peer: Peer: permanent_uuid: "def016287d2a48c5b59d552164d1e243" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:35.592888 13454 mvcc.cc:205] Tried to move back new op lower bound from 7298355509614821376 to 7298355509583368192. Current Snapshot: MvccSnapshot[applied={T|T < 7298355509614821376}]
I20260618 23:24:35.593528 13455 mvcc.cc:205] Tried to move back new op lower bound from 7298355509614821376 to 7298355509583368192. Current Snapshot: MvccSnapshot[applied={T|T < 7298355509614821376 or (T in {7298355509614821376})}]
I20260618 23:24:35.594753 12975 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:24:35.594836 12975 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39592:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:24:35.595535 12975 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 29958908d79f4e99981793a688a77cbc: Sending DeleteTablet for 3 replicas of tablet c23265c87f4c46339d5ba1a8453acea9
I20260618 23:24:35.595877 13098 tablet_service.cc:1514] Processing DeleteTablet for tablet c23265c87f4c46339d5ba1a8453acea9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:35 UTC) from {username='slave'} at 127.0.0.1:51646
I20260618 23:24:35.595893 13360 tablet_service.cc:1514] Processing DeleteTablet for tablet c23265c87f4c46339d5ba1a8453acea9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:35 UTC) from {username='slave'} at 127.0.0.1:44126
I20260618 23:24:35.596095 13463 tablet_replica.cc:330] T c23265c87f4c46339d5ba1a8453acea9 P 37ee37589b6b43c5a3b2efe7c10b6a50: stopping tablet replica
I20260618 23:24:35.596081 13229 tablet_service.cc:1514] Processing DeleteTablet for tablet c23265c87f4c46339d5ba1a8453acea9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:35 UTC) from {username='slave'} at 127.0.0.1:51452
I20260618 23:24:35.596184 13463 raft_consensus.cc:2229] T c23265c87f4c46339d5ba1a8453acea9 P 37ee37589b6b43c5a3b2efe7c10b6a50 [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:24:35.596302 13463 raft_consensus.cc:2258] T c23265c87f4c46339d5ba1a8453acea9 P 37ee37589b6b43c5a3b2efe7c10b6a50 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:35.596378 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 13036
I20260618 23:24:35.602393 13464 tablet_replica.cc:330] T c23265c87f4c46339d5ba1a8453acea9 P daec3f238f514f0e84a213fffb6ba49c: stopping tablet replica
I20260618 23:24:35.602516 13464 raft_consensus.cc:2229] T c23265c87f4c46339d5ba1a8453acea9 P daec3f238f514f0e84a213fffb6ba49c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:24:35.602623 13464 raft_consensus.cc:2258] T c23265c87f4c46339d5ba1a8453acea9 P daec3f238f514f0e84a213fffb6ba49c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:35.602901 13465 tablet_replica.cc:330] T c23265c87f4c46339d5ba1a8453acea9 P def016287d2a48c5b59d552164d1e243: stopping tablet replica
I20260618 23:24:35.603003 13465 raft_consensus.cc:2229] T c23265c87f4c46339d5ba1a8453acea9 P def016287d2a48c5b59d552164d1e243 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:24:35.603106 13465 raft_consensus.cc:2258] T c23265c87f4c46339d5ba1a8453acea9 P def016287d2a48c5b59d552164d1e243 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:35.603240 13464 ts_tablet_manager.cc:1905] T c23265c87f4c46339d5ba1a8453acea9 P daec3f238f514f0e84a213fffb6ba49c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260618 23:24:35.603302 12963 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.39.129:38429 (error 108)
I20260618 23:24:35.603598 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 13167
I20260618 23:24:35.605674 13464 ts_tablet_manager.cc:1918] T c23265c87f4c46339d5ba1a8453acea9 P daec3f238f514f0e84a213fffb6ba49c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:24:35.605777 13464 log.cc:1196] T c23265c87f4c46339d5ba1a8453acea9 P daec3f238f514f0e84a213fffb6ba49c: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781825053542692-8350-0/minicluster-data/ts-2/wal/wals/c23265c87f4c46339d5ba1a8453acea9
I20260618 23:24:35.606029 13464 ts_tablet_manager.cc:1939] T c23265c87f4c46339d5ba1a8453acea9 P daec3f238f514f0e84a213fffb6ba49c: Deleting consensus metadata
I20260618 23:24:35.606433 12960 catalog_manager.cc:4841] TS daec3f238f514f0e84a213fffb6ba49c (127.8.39.131:41741): tablet c23265c87f4c46339d5ba1a8453acea9 (table test-table [id=31e9b4542f384b1798a5209518c7f49e]) successfully deleted
I20260618 23:24:35.607082 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:35.606957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:38429 (local address 127.0.0.1:51678)
0618 23:24:35.607017 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:35.607058 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.129:38429: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260618 23:24:35.607203 12963 catalog_manager.cc:4584] TS 37ee37589b6b43c5a3b2efe7c10b6a50 (127.8.39.129:38429): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c23265c87f4c46339d5ba1a8453acea9: Network error: Client connection negotiation failed: client connection to 127.8.39.129:38429: connect: Connection refused (error 111)
I20260618 23:24:35.610486 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 13298
I20260618 23:24:35.610494 13434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:35.610360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:46371 (local address 127.0.0.1:51462)
0618 23:24:35.610427 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:35.610454 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.130:46371: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260618 23:24:35.615609 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 12945
2026-06-18T23:24:35Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (582 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260618 23:24:35.634821 8350 mini_oidc.cc:152] Starting JWKS server
I20260618 23:24:35.635771 8350 webserver.cc:460] Webserver started at https://127.0.0.1:40955/ using document root <none> and password file <none>
I20260618 23:24:35.635885 8350 mini_oidc.cc:189] Starting OIDC Discovery server
I20260618 23:24:35.636071 8350 webserver.cc:460] Webserver started at http://127.0.0.1:42949/ using document root <none> and password file <none>
2026-06-18T23:24:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:24:35Z Disabled control of system clock
I20260618 23:24:35.643143 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:41849
--webserver_interface=127.8.39.190
--webserver_port=0
--builtin_ntp_servers=127.8.39.148:35143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:41849
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40955/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825053542692-8350-0/testchainca.cert with env {}
W20260618 23:24:35.721055 13472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:35.721258 13472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:35.721279 13472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:35.722347 13472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:24:35.722381 13472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:35.722394 13472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:35.722407 13472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:24:35.722419 13472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260618 23:24:35.722432 13472 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260618 23:24:35.722443 13472 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40955/jwks.json
I20260618 23:24:35.723837 13472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:35143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:41849
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:41849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40955/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825053542692-8350-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:35.724007 13472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:35.724164 13472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:35.728791 13477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:35.728847 13472 server_base.cc:1034] running on GCE node
W20260618 23:24:35.728791 13480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:35.728816 13478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:35.729315 13472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:35.729585 13472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:35.730763 13472 hybrid_clock.cc:648] HybridClock initialized: now 1781825075730737 us; error 34 us; skew 500 ppm
I20260618 23:24:35.731973 13472 webserver.cc:460] Webserver started at http://127.8.39.190:34929/ using document root <none> and password file <none>
I20260618 23:24:35.732203 13472 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:35.732286 13472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:35.732434 13472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:35.733515 13472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "4bb868d2be0e402bab5ab05357e7e024"
format_stamp: "Formatted at 2026-06-18 23:24:35 on dist-test-slave-47qt"
server_key: "57cd58ec94d54efd1fb922169f704d0d"
server_key_iv: "f54c922477b77e77c479722ba6804b29"
server_key_version: "encryptionkey@0"
I20260618 23:24:35.733860 13472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "4bb868d2be0e402bab5ab05357e7e024"
format_stamp: "Formatted at 2026-06-18 23:24:35 on dist-test-slave-47qt"
server_key: "57cd58ec94d54efd1fb922169f704d0d"
server_key_iv: "f54c922477b77e77c479722ba6804b29"
server_key_version: "encryptionkey@0"
I20260618 23:24:35.735100 13472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:35.735808 13486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:35.735977 13472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260618 23:24:35.736078 13472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "4bb868d2be0e402bab5ab05357e7e024"
format_stamp: "Formatted at 2026-06-18 23:24:35 on dist-test-slave-47qt"
server_key: "57cd58ec94d54efd1fb922169f704d0d"
server_key_iv: "f54c922477b77e77c479722ba6804b29"
server_key_version: "encryptionkey@0"
I20260618 23:24:35.736148 13472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:35.757661 13472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:35.757967 13472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:35.758118 13472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:35.761878 13472 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:41849
I20260618 23:24:35.761984 13538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:41849 every 8 connection(s)
I20260618 23:24:35.762288 13472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:24:35.763223 13539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:35.765345 13539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bb868d2be0e402bab5ab05357e7e024: Bootstrap starting.
I20260618 23:24:35.765847 13539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bb868d2be0e402bab5ab05357e7e024: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:35.766081 13539 log.cc:824] T 00000000000000000000000000000000 P 4bb868d2be0e402bab5ab05357e7e024: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:35.766697 13539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bb868d2be0e402bab5ab05357e7e024: No bootstrap required, opened a new log
I20260618 23:24:35.767280 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 13472
I20260618 23:24:35.767390 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:24:35.767554 8350 external_mini_cluster.cc:1428] Setting key 7de772c6beff64d73593083cb55a6727
I20260618 23:24:35.768903 13539 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4bb868d2be0e402bab5ab05357e7e024 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb868d2be0e402bab5ab05357e7e024" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41849 } }
I20260618 23:24:35.769071 13539 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4bb868d2be0e402bab5ab05357e7e024 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:35.769110 13539 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4bb868d2be0e402bab5ab05357e7e024 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bb868d2be0e402bab5ab05357e7e024, State: Initialized, Role: FOLLOWER
I20260618 23:24:35.769264 13539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bb868d2be0e402bab5ab05357e7e024 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb868d2be0e402bab5ab05357e7e024" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41849 } }
I20260618 23:24:35.769352 13539 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4bb868d2be0e402bab5ab05357e7e024 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:24:35.769418 13539 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4bb868d2be0e402bab5ab05357e7e024 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:24:35.769496 13539 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4bb868d2be0e402bab5ab05357e7e024 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:35.770130 13539 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4bb868d2be0e402bab5ab05357e7e024 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb868d2be0e402bab5ab05357e7e024" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41849 } }
I20260618 23:24:35.770253 13539 leader_election.cc:302] T 00000000000000000000000000000000 P 4bb868d2be0e402bab5ab05357e7e024 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4bb868d2be0e402bab5ab05357e7e024; no voters:
I20260618 23:24:35.770448 13539 leader_election.cc:288] T 00000000000000000000000000000000 P 4bb868d2be0e402bab5ab05357e7e024 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:24:35.770481 13544 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4bb868d2be0e402bab5ab05357e7e024 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:35.770592 13544 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4bb868d2be0e402bab5ab05357e7e024 [term 1 LEADER]: Becoming Leader. State: Replica: 4bb868d2be0e402bab5ab05357e7e024, State: Running, Role: LEADER
I20260618 23:24:35.770671 13544 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bb868d2be0e402bab5ab05357e7e024 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb868d2be0e402bab5ab05357e7e024" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41849 } }
I20260618 23:24:35.770712 13539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4bb868d2be0e402bab5ab05357e7e024 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:24:35.770992 13546 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bb868d2be0e402bab5ab05357e7e024 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bb868d2be0e402bab5ab05357e7e024. Latest consensus state: current_term: 1 leader_uuid: "4bb868d2be0e402bab5ab05357e7e024" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb868d2be0e402bab5ab05357e7e024" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41849 } } }
I20260618 23:24:35.771219 13546 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bb868d2be0e402bab5ab05357e7e024 [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:35.771158 13545 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bb868d2be0e402bab5ab05357e7e024 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4bb868d2be0e402bab5ab05357e7e024" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4bb868d2be0e402bab5ab05357e7e024" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41849 } } }
I20260618 23:24:35.771456 13545 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bb868d2be0e402bab5ab05357e7e024 [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:35.771672 13553 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:24:35.772197 13553 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:24:35.774279 13543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:35.768065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59592 (local address 127.8.39.190:41849)
0618 23:24:35.768258 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:35.768265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:35.768288 (+ 23us) server_negotiation.cc:407] Connection header received
0618 23:24:35.768887 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:35.768894 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:35.768950 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:35.769060 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:35.769363 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:35.770842 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:35.771925 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:35.772575 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:35.772707 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:35.773395 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:35.773398 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:35.773399 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:24:35.773598 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:35.773642 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:24:35.773656 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:35.773723 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:35.773983 (+ 260us) server_negotiation.cc:299] Negotiation successful
0618 23:24:35.774056 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
I20260618 23:24:35.774567 13553 catalog_manager.cc:1347] Generated new cluster ID: d622cf49820e4af0a3a5d18f2146c9a5
I20260618 23:24:35.774614 13553 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:24:35.775094 8350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:24:35.794508 13553 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:24:35.795228 13553 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:24:35.809571 13553 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4bb868d2be0e402bab5ab05357e7e024: Generated new TSK 0
I20260618 23:24:35.809804 13553 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260618 23:24:35.840868 13569 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260618 23:24:35.840970 13569 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
W20260618 23:24:35.841492 13568 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:35.817850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41849 (local address 127.0.0.1:59594)
0618 23:24:35.818050 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:35.818062 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:35.818138 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:35.818333 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:35.818334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:35.818336 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0618 23:24:35.818402 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:35.818404 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:35.819088 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:35.819089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:35.820065 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:35.820068 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:35.820430 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:35.820431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:35.820495 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:35.820517 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0618 23:24:35.841094 (+ 20577us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40955/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0618 23:24:35.841254 (+ 160us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.39.190:41849: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40955/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":112,"threads_started":1}
I20260618 23:24:35.841800 13543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:35.817958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59594 (local address 127.8.39.190:41849)
0618 23:24:35.818008 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:35.818010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:35.818129 (+ 119us) server_negotiation.cc:407] Connection header received
0618 23:24:35.818244 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:35.818245 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:35.818256 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:35.818308 (+ 52us) server_negotiation.cc:226] Negotiated authn=JWT
0618 23:24:35.818498 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:35.819009 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:35.820160 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:35.820349 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:35.820413 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:35.820675 (+ 262us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0618 23:24:35.840878 (+ 20203us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40955/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0618 23:24:35.841742 (+ 864us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59594: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40955/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":27}
W20260618 23:24:35.841845 13543 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59594: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40955/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.8.39.190:41849: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40955/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260618 23:24:35.841953 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 13472
2026-06-18T23:24:35Z chronyd exiting
I20260618 23:24:35.878161 8350 test_util.cc:175] -----------------------------------------------
I20260618 23:24:35.878237 8350 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781825053542692-8350-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (244 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260618 23:24:35.879009 8350 mini_oidc.cc:152] Starting JWKS server
I20260618 23:24:35.879698 8350 webserver.cc:460] Webserver started at https://127.0.0.1:41171/ using document root <none> and password file <none>
I20260618 23:24:35.879767 8350 mini_oidc.cc:189] Starting OIDC Discovery server
I20260618 23:24:35.879937 8350 webserver.cc:460] Webserver started at http://127.0.0.1:34605/ using document root <none> and password file <none>
2026-06-18T23:24:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:24:35Z Disabled control of system clock
I20260618 23:24:35.886900 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:36709
--webserver_interface=127.8.39.190
--webserver_port=0
--builtin_ntp_servers=127.8.39.148:44583
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:36709
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41171/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825053542692-8350-0/testchainca.cert with env {}
W20260618 23:24:35.966948 13575 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:35.967185 13575 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:35.967219 13575 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:35.968322 13575 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:24:35.968392 13575 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:35.968416 13575 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:35.968444 13575 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:24:35.968477 13575 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260618 23:24:35.968499 13575 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260618 23:24:35.968523 13575 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41171/jwks.json
I20260618 23:24:35.970009 13575 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:44583
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:36709
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:36709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41171/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825053542692-8350-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13575
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:35.970273 13575 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:35.970522 13575 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:35.975591 13581 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:35.975605 13583 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:35.975605 13580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:35.976073 13575 server_base.cc:1034] running on GCE node
I20260618 23:24:35.976316 13575 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:35.976585 13575 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:35.977752 13575 hybrid_clock.cc:648] HybridClock initialized: now 1781825075977752 us; error 42 us; skew 500 ppm
I20260618 23:24:35.978992 13575 webserver.cc:460] Webserver started at http://127.8.39.190:45269/ using document root <none> and password file <none>
I20260618 23:24:35.979234 13575 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:35.979301 13575 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:35.979463 13575 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:35.980500 13575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "cda669ccd1fd4b50a71fda9dba5ae968"
format_stamp: "Formatted at 2026-06-18 23:24:35 on dist-test-slave-47qt"
server_key: "573a1756346dba55eeb1725694967e6c"
server_key_iv: "5281b50fc272223f813ff2f3c63256d8"
server_key_version: "encryptionkey@0"
I20260618 23:24:35.980865 13575 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "cda669ccd1fd4b50a71fda9dba5ae968"
format_stamp: "Formatted at 2026-06-18 23:24:35 on dist-test-slave-47qt"
server_key: "573a1756346dba55eeb1725694967e6c"
server_key_iv: "5281b50fc272223f813ff2f3c63256d8"
server_key_version: "encryptionkey@0"
I20260618 23:24:35.982311 13575 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:24:35.983219 13589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:35.983546 13575 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:24:35.983656 13575 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "cda669ccd1fd4b50a71fda9dba5ae968"
format_stamp: "Formatted at 2026-06-18 23:24:35 on dist-test-slave-47qt"
server_key: "573a1756346dba55eeb1725694967e6c"
server_key_iv: "5281b50fc272223f813ff2f3c63256d8"
server_key_version: "encryptionkey@0"
I20260618 23:24:35.983742 13575 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:35.996210 13575 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:35.996521 13575 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:35.996672 13575 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:36.000655 13575 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:36709
I20260618 23:24:36.000800 13641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:36709 every 8 connection(s)
I20260618 23:24:36.001060 13575 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:24:36.002074 13642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:36.004302 13642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda669ccd1fd4b50a71fda9dba5ae968: Bootstrap starting.
I20260618 23:24:36.004878 13642 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cda669ccd1fd4b50a71fda9dba5ae968: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:36.005231 13642 log.cc:824] T 00000000000000000000000000000000 P cda669ccd1fd4b50a71fda9dba5ae968: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:36.005950 13642 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cda669ccd1fd4b50a71fda9dba5ae968: No bootstrap required, opened a new log
I20260618 23:24:36.007534 13642 raft_consensus.cc:348] T 00000000000000000000000000000000 P cda669ccd1fd4b50a71fda9dba5ae968 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda669ccd1fd4b50a71fda9dba5ae968" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 36709 } }
I20260618 23:24:36.007671 13642 raft_consensus.cc:374] T 00000000000000000000000000000000 P cda669ccd1fd4b50a71fda9dba5ae968 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:36.007694 13642 raft_consensus.cc:729] T 00000000000000000000000000000000 P cda669ccd1fd4b50a71fda9dba5ae968 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cda669ccd1fd4b50a71fda9dba5ae968, State: Initialized, Role: FOLLOWER
I20260618 23:24:36.007862 13642 consensus_queue.cc:260] T 00000000000000000000000000000000 P cda669ccd1fd4b50a71fda9dba5ae968 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda669ccd1fd4b50a71fda9dba5ae968" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 36709 } }
I20260618 23:24:36.007932 13642 raft_consensus.cc:388] T 00000000000000000000000000000000 P cda669ccd1fd4b50a71fda9dba5ae968 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:24:36.007958 13642 raft_consensus.cc:482] T 00000000000000000000000000000000 P cda669ccd1fd4b50a71fda9dba5ae968 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:24:36.007999 13642 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cda669ccd1fd4b50a71fda9dba5ae968 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:36.008549 13642 raft_consensus.cc:504] T 00000000000000000000000000000000 P cda669ccd1fd4b50a71fda9dba5ae968 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda669ccd1fd4b50a71fda9dba5ae968" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 36709 } }
I20260618 23:24:36.008633 13642 leader_election.cc:302] T 00000000000000000000000000000000 P cda669ccd1fd4b50a71fda9dba5ae968 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cda669ccd1fd4b50a71fda9dba5ae968; no voters:
I20260618 23:24:36.008821 13642 leader_election.cc:288] T 00000000000000000000000000000000 P cda669ccd1fd4b50a71fda9dba5ae968 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:24:36.008981 13646 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cda669ccd1fd4b50a71fda9dba5ae968 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:36.009078 13642 sys_catalog.cc:564] T 00000000000000000000000000000000 P cda669ccd1fd4b50a71fda9dba5ae968 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:24:36.009101 13646 raft_consensus.cc:686] T 00000000000000000000000000000000 P cda669ccd1fd4b50a71fda9dba5ae968 [term 1 LEADER]: Becoming Leader. State: Replica: cda669ccd1fd4b50a71fda9dba5ae968, State: Running, Role: LEADER
I20260618 23:24:36.009223 13646 consensus_queue.cc:237] T 00000000000000000000000000000000 P cda669ccd1fd4b50a71fda9dba5ae968 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda669ccd1fd4b50a71fda9dba5ae968" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 36709 } }
I20260618 23:24:36.009624 13647 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda669ccd1fd4b50a71fda9dba5ae968 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cda669ccd1fd4b50a71fda9dba5ae968. Latest consensus state: current_term: 1 leader_uuid: "cda669ccd1fd4b50a71fda9dba5ae968" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda669ccd1fd4b50a71fda9dba5ae968" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 36709 } } }
I20260618 23:24:36.009718 13647 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda669ccd1fd4b50a71fda9dba5ae968 [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:36.009738 13645 sys_catalog.cc:455] T 00000000000000000000000000000000 P cda669ccd1fd4b50a71fda9dba5ae968 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cda669ccd1fd4b50a71fda9dba5ae968" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cda669ccd1fd4b50a71fda9dba5ae968" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 36709 } } }
I20260618 23:24:36.009836 13645 sys_catalog.cc:458] T 00000000000000000000000000000000 P cda669ccd1fd4b50a71fda9dba5ae968 [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:36.009955 13654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:24:36.010474 13654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:24:36.011117 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 13575
I20260618 23:24:36.011229 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:24:36.011400 8350 external_mini_cluster.cc:1428] Setting key 7d103d7c1e47907fc49b587cbebc5446
I20260618 23:24:36.012332 13654 catalog_manager.cc:1347] Generated new cluster ID: cc83e2f5857c432b842b2df69d9df8bf
I20260618 23:24:36.012387 13654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:24:36.018249 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:36.011881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52748 (local address 127.8.39.190:36709)
0618 23:24:36.012082 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:36.012091 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:36.012121 (+ 30us) server_negotiation.cc:407] Connection header received
0618 23:24:36.012232 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:36.012239 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:36.012304 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:36.012449 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:36.012763 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:36.014189 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:36.015344 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:36.016010 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:36.016139 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:36.017229 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:36.017233 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:36.017234 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:24:36.017436 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:36.017477 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:24:36.017491 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:36.017558 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:36.017817 (+ 259us) server_negotiation.cc:299] Negotiation successful
0618 23:24:36.017901 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
I20260618 23:24:36.020045 8350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:24:36.033694 13654 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:24:36.034412 13654 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:24:36.050802 13654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cda669ccd1fd4b50a71fda9dba5ae968: Generated new TSK 0
I20260618 23:24:36.051031 13654 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260618 23:24:36.072551 13672 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260618 23:24:36.072634 13672 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260618 23:24:36.072932 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:36.057929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52750 (local address 127.8.39.190:36709)
0618 23:24:36.057976 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:36.057980 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:36.058052 (+ 72us) server_negotiation.cc:407] Connection header received
0618 23:24:36.058139 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:36.058139 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:36.058149 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:36.058203 (+ 54us) server_negotiation.cc:226] Negotiated authn=JWT
0618 23:24:36.058321 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:36.058996 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:36.060191 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:36.060401 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:36.060462 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:36.060641 (+ 179us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0618 23:24:36.072767 (+ 12126us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41171/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0618 23:24:36.072883 (+ 116us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52750: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41171/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":24}
W20260618 23:24:36.072999 13665 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52750: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41171/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260618 23:24:36.073093 13671 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:36.057821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:36709 (local address 127.0.0.1:52750)
0618 23:24:36.057992 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:36.058003 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:36.058071 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:36.058211 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:36.058212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:36.058214 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0618 23:24:36.058267 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:36.058269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:36.059140 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:36.059141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:36.060079 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:36.060081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:36.060479 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:36.060480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:36.060546 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:36.060551 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0618 23:24:36.072871 (+ 12320us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41171/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0618 23:24:36.072995 (+ 124us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.39.190:36709: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41171/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":106,"threads_started":1}
I20260618 23:24:36.073421 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 13575
2026-06-18T23:24:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (232 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260618 23:24:36.111644 8350 mini_oidc.cc:152] Starting JWKS server
I20260618 23:24:36.112393 8350 webserver.cc:460] Webserver started at https://127.0.0.1:36013/ using document root <none> and password file <none>
I20260618 23:24:36.112468 8350 mini_oidc.cc:189] Starting OIDC Discovery server
I20260618 23:24:36.112643 8350 webserver.cc:460] Webserver started at http://127.0.0.1:40951/ using document root <none> and password file <none>
2026-06-18T23:24:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:24:36Z Disabled control of system clock
I20260618 23:24:36.122149 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:46425
--webserver_interface=127.8.39.190
--webserver_port=0
--builtin_ntp_servers=127.8.39.148:40375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:46425
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36013/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260618 23:24:36.202095 13678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:36.202292 13678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:36.202313 13678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:36.203447 13678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:24:36.203487 13678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:36.203500 13678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:36.203513 13678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:24:36.203526 13678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260618 23:24:36.203575 13678 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260618 23:24:36.203591 13678 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36013/jwks.json
I20260618 23:24:36.204928 13678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:40375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:46425
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:46425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36013/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13678
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:36.205196 13678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:36.205466 13678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:36.210677 13683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:36.210668 13686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:36.210668 13684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:36.211036 13678 server_base.cc:1034] running on GCE node
I20260618 23:24:36.211246 13678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:36.211472 13678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:36.212636 13678 hybrid_clock.cc:648] HybridClock initialized: now 1781825076212619 us; error 28 us; skew 500 ppm
I20260618 23:24:36.214008 13678 webserver.cc:460] Webserver started at http://127.8.39.190:43469/ using document root <none> and password file <none>
I20260618 23:24:36.214272 13678 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:36.214321 13678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:36.214454 13678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:36.215463 13678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "0094a4d56ba84465a9eeab9cada95384"
format_stamp: "Formatted at 2026-06-18 23:24:36 on dist-test-slave-47qt"
server_key: "1d638319b3d9b55d4ce95c2e55fcf937"
server_key_iv: "a6ce519112931c9decc74f1226c3c085"
server_key_version: "encryptionkey@0"
I20260618 23:24:36.215812 13678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "0094a4d56ba84465a9eeab9cada95384"
format_stamp: "Formatted at 2026-06-18 23:24:36 on dist-test-slave-47qt"
server_key: "1d638319b3d9b55d4ce95c2e55fcf937"
server_key_iv: "a6ce519112931c9decc74f1226c3c085"
server_key_version: "encryptionkey@0"
I20260618 23:24:36.217039 13678 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:24:36.217782 13692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:36.217960 13678 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260618 23:24:36.218046 13678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "0094a4d56ba84465a9eeab9cada95384"
format_stamp: "Formatted at 2026-06-18 23:24:36 on dist-test-slave-47qt"
server_key: "1d638319b3d9b55d4ce95c2e55fcf937"
server_key_iv: "a6ce519112931c9decc74f1226c3c085"
server_key_version: "encryptionkey@0"
I20260618 23:24:36.218120 13678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:36.236054 13678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:36.236388 13678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:36.236547 13678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:36.240525 13678 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:46425
I20260618 23:24:36.240559 13744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:46425 every 8 connection(s)
I20260618 23:24:36.240906 13678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:24:36.241724 13745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:36.244406 13745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0094a4d56ba84465a9eeab9cada95384: Bootstrap starting.
I20260618 23:24:36.244977 13745 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0094a4d56ba84465a9eeab9cada95384: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:36.245296 13745 log.cc:824] T 00000000000000000000000000000000 P 0094a4d56ba84465a9eeab9cada95384: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:36.245968 13745 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0094a4d56ba84465a9eeab9cada95384: No bootstrap required, opened a new log
I20260618 23:24:36.247184 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 13678
I20260618 23:24:36.247282 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:24:36.247466 8350 external_mini_cluster.cc:1428] Setting key 3749a93399f39f7766c376047fd6d31d
I20260618 23:24:36.247929 13745 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0094a4d56ba84465a9eeab9cada95384 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0094a4d56ba84465a9eeab9cada95384" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 46425 } }
I20260618 23:24:36.248111 13745 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0094a4d56ba84465a9eeab9cada95384 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:36.248193 13745 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0094a4d56ba84465a9eeab9cada95384 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0094a4d56ba84465a9eeab9cada95384, State: Initialized, Role: FOLLOWER
I20260618 23:24:36.248318 13745 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0094a4d56ba84465a9eeab9cada95384 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0094a4d56ba84465a9eeab9cada95384" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 46425 } }
I20260618 23:24:36.248400 13745 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0094a4d56ba84465a9eeab9cada95384 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:24:36.248443 13745 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0094a4d56ba84465a9eeab9cada95384 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:24:36.248502 13745 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0094a4d56ba84465a9eeab9cada95384 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:36.249431 13745 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0094a4d56ba84465a9eeab9cada95384 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0094a4d56ba84465a9eeab9cada95384" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 46425 } }
I20260618 23:24:36.249552 13745 leader_election.cc:302] T 00000000000000000000000000000000 P 0094a4d56ba84465a9eeab9cada95384 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0094a4d56ba84465a9eeab9cada95384; no voters:
I20260618 23:24:36.249718 13745 leader_election.cc:288] T 00000000000000000000000000000000 P 0094a4d56ba84465a9eeab9cada95384 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:24:36.249758 13750 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0094a4d56ba84465a9eeab9cada95384 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:36.249877 13750 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0094a4d56ba84465a9eeab9cada95384 [term 1 LEADER]: Becoming Leader. State: Replica: 0094a4d56ba84465a9eeab9cada95384, State: Running, Role: LEADER
I20260618 23:24:36.249970 13745 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0094a4d56ba84465a9eeab9cada95384 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:24:36.249955 13750 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0094a4d56ba84465a9eeab9cada95384 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0094a4d56ba84465a9eeab9cada95384" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 46425 } }
I20260618 23:24:36.251252 13751 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0094a4d56ba84465a9eeab9cada95384 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0094a4d56ba84465a9eeab9cada95384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0094a4d56ba84465a9eeab9cada95384" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 46425 } } }
I20260618 23:24:36.251335 13751 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0094a4d56ba84465a9eeab9cada95384 [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:36.251412 13752 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0094a4d56ba84465a9eeab9cada95384 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0094a4d56ba84465a9eeab9cada95384. Latest consensus state: current_term: 1 leader_uuid: "0094a4d56ba84465a9eeab9cada95384" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0094a4d56ba84465a9eeab9cada95384" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 46425 } } }
I20260618 23:24:36.251492 13752 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0094a4d56ba84465a9eeab9cada95384 [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:36.251549 13765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:24:36.252156 13765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:24:36.253099 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:36.247972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56062 (local address 127.8.39.190:46425)
0618 23:24:36.248218 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:36.248227 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:36.248260 (+ 33us) server_negotiation.cc:407] Connection header received
0618 23:24:36.248372 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:36.248381 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:36.248469 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:36.248598 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:36.248798 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:36.249982 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:36.250790 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:36.251426 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:36.251546 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:36.252442 (+ 896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:36.252446 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:36.252447 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:24:36.252617 (+ 170us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:36.252648 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:24:36.252658 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:36.252740 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:36.252813 (+ 73us) server_negotiation.cc:299] Negotiation successful
0618 23:24:36.252888 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":94,"threads_started":1}
I20260618 23:24:36.253679 13765 catalog_manager.cc:1347] Generated new cluster ID: ab1d2691d7d94ce6883343a80e0d3d8e
I20260618 23:24:36.253731 13765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:24:36.254122 8350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:24:36.295483 13765 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:24:36.296221 13765 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:24:36.303895 13765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0094a4d56ba84465a9eeab9cada95384: Generated new TSK 0
I20260618 23:24:36.304113 13765 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:24:36.324452 13749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:36.320935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56078 (local address 127.8.39.190:46425)
0618 23:24:36.320996 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:36.320998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:36.321102 (+ 104us) server_negotiation.cc:407] Connection header received
0618 23:24:36.321208 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:36.321209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:36.321222 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:36.321293 (+ 71us) server_negotiation.cc:226] Negotiated authn=JWT
0618 23:24:36.321464 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:36.322032 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:36.323186 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:36.323365 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:36.323422 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:36.323639 (+ 217us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0618 23:24:36.324301 (+ 662us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36013/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0618 23:24:36.324394 (+ 93us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56078: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36013/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":38}
W20260618 23:24:36.324537 13749 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56078: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36013/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260618 23:24:36.324615 13774 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:36.320840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:46425 (local address 127.0.0.1:56078)
0618 23:24:36.321023 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:36.321035 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:36.321108 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:36.321315 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:36.321317 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:36.321318 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0618 23:24:36.321392 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:36.321395 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:36.322121 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:36.322122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:36.323090 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:36.323093 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:36.323459 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:36.323460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:36.323524 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:36.323531 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0618 23:24:36.324400 (+ 869us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36013/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0618 23:24:36.324509 (+ 109us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.39.190:46425: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36013/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":97,"threads_started":1}
I20260618 23:24:36.324786 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 13678
E20260618 23:24:36.336124 13775 webserver.cc:533] Webserver: sslize failed:
E20260618 23:24:36.336205 13775 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-18T23:24:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (251 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-18T23:24:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:24:36Z Disabled control of system clock
I20260618 23:24:36.369985 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:39745
--webserver_interface=127.8.39.190
--webserver_port=0
--builtin_ntp_servers=127.8.39.148:37273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:39745
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825053542692-8350-0/testchainca.cert with env {}
W20260618 23:24:36.450291 13779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:36.450539 13779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:36.450578 13779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:36.451783 13779 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:24:36.451869 13779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:36.451895 13779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:36.451923 13779 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:24:36.451956 13779 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260618 23:24:36.451982 13779 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260618 23:24:36.452011 13779 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260618 23:24:36.453660 13779 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:37273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:39745
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:39745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825053542692-8350-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:36.453939 13779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:36.454154 13779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:36.458993 13784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:36.459003 13787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:36.459051 13785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:36.459232 13779 server_base.cc:1034] running on GCE node
I20260618 23:24:36.459444 13779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:36.459682 13779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:36.460831 13779 hybrid_clock.cc:648] HybridClock initialized: now 1781825076460817 us; error 28 us; skew 500 ppm
I20260618 23:24:36.462078 13779 webserver.cc:460] Webserver started at http://127.8.39.190:36639/ using document root <none> and password file <none>
I20260618 23:24:36.462329 13779 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:36.462378 13779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:36.462505 13779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:36.463676 13779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "6afeb1878aeb488389ef6ab7eb5e007d"
format_stamp: "Formatted at 2026-06-18 23:24:36 on dist-test-slave-47qt"
server_key: "8e8b56f51212789028e495ab8ae316ed"
server_key_iv: "6d3574d26f31c78e3c2508e2f5002318"
server_key_version: "encryptionkey@0"
I20260618 23:24:36.464074 13779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "6afeb1878aeb488389ef6ab7eb5e007d"
format_stamp: "Formatted at 2026-06-18 23:24:36 on dist-test-slave-47qt"
server_key: "8e8b56f51212789028e495ab8ae316ed"
server_key_iv: "6d3574d26f31c78e3c2508e2f5002318"
server_key_version: "encryptionkey@0"
I20260618 23:24:36.465418 13779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:24:36.466149 13793 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:36.466359 13779 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:36.466427 13779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "6afeb1878aeb488389ef6ab7eb5e007d"
format_stamp: "Formatted at 2026-06-18 23:24:36 on dist-test-slave-47qt"
server_key: "8e8b56f51212789028e495ab8ae316ed"
server_key_iv: "6d3574d26f31c78e3c2508e2f5002318"
server_key_version: "encryptionkey@0"
I20260618 23:24:36.466516 13779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:36.477555 13779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:36.477841 13779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:36.477991 13779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:36.481640 13779 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:39745
I20260618 23:24:36.481871 13845 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:39745 every 8 connection(s)
I20260618 23:24:36.482085 13779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:24:36.483448 13846 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:36.484226 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 13779
I20260618 23:24:36.484370 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:24:36.484687 8350 external_mini_cluster.cc:1428] Setting key a4a17cdf383852ba02cebf81a0c93cc7
I20260618 23:24:36.486289 13846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6afeb1878aeb488389ef6ab7eb5e007d: Bootstrap starting.
I20260618 23:24:36.486984 13846 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6afeb1878aeb488389ef6ab7eb5e007d: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:36.487309 13846 log.cc:824] T 00000000000000000000000000000000 P 6afeb1878aeb488389ef6ab7eb5e007d: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:36.488025 13846 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6afeb1878aeb488389ef6ab7eb5e007d: No bootstrap required, opened a new log
I20260618 23:24:36.490015 13846 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6afeb1878aeb488389ef6ab7eb5e007d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6afeb1878aeb488389ef6ab7eb5e007d" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 39745 } }
I20260618 23:24:36.490186 13846 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6afeb1878aeb488389ef6ab7eb5e007d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:36.490223 13846 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6afeb1878aeb488389ef6ab7eb5e007d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6afeb1878aeb488389ef6ab7eb5e007d, State: Initialized, Role: FOLLOWER
I20260618 23:24:36.490363 13846 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6afeb1878aeb488389ef6ab7eb5e007d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6afeb1878aeb488389ef6ab7eb5e007d" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 39745 } }
I20260618 23:24:36.490463 13846 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6afeb1878aeb488389ef6ab7eb5e007d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:24:36.490499 13846 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6afeb1878aeb488389ef6ab7eb5e007d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:24:36.490549 13846 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6afeb1878aeb488389ef6ab7eb5e007d [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:36.491146 13846 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6afeb1878aeb488389ef6ab7eb5e007d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6afeb1878aeb488389ef6ab7eb5e007d" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 39745 } }
I20260618 23:24:36.491252 13846 leader_election.cc:302] T 00000000000000000000000000000000 P 6afeb1878aeb488389ef6ab7eb5e007d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6afeb1878aeb488389ef6ab7eb5e007d; no voters:
I20260618 23:24:36.491408 13846 leader_election.cc:288] T 00000000000000000000000000000000 P 6afeb1878aeb488389ef6ab7eb5e007d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:24:36.491465 13851 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6afeb1878aeb488389ef6ab7eb5e007d [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:36.491587 13851 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6afeb1878aeb488389ef6ab7eb5e007d [term 1 LEADER]: Becoming Leader. State: Replica: 6afeb1878aeb488389ef6ab7eb5e007d, State: Running, Role: LEADER
I20260618 23:24:36.491686 13851 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6afeb1878aeb488389ef6ab7eb5e007d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6afeb1878aeb488389ef6ab7eb5e007d" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 39745 } }
I20260618 23:24:36.491670 13846 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6afeb1878aeb488389ef6ab7eb5e007d [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:24:36.492035 13849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:36.485412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40408 (local address 127.8.39.190:39745)
0618 23:24:36.485636 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:36.485645 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:36.485677 (+ 32us) server_negotiation.cc:407] Connection header received
0618 23:24:36.486178 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:36.486187 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:36.486259 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:36.486411 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:36.486726 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:36.488025 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:36.489394 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:36.490058 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:36.490199 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:36.490980 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:36.490984 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:36.490984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:24:36.491201 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:36.491243 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:24:36.491257 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:36.491327 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:36.491477 (+ 150us) server_negotiation.cc:299] Negotiation successful
0618 23:24:36.491567 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":80,"threads_started":1}
I20260618 23:24:36.492671 13851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6afeb1878aeb488389ef6ab7eb5e007d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6afeb1878aeb488389ef6ab7eb5e007d. Latest consensus state: current_term: 1 leader_uuid: "6afeb1878aeb488389ef6ab7eb5e007d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6afeb1878aeb488389ef6ab7eb5e007d" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 39745 } } }
I20260618 23:24:36.492942 8350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:24:36.493458 13853 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6afeb1878aeb488389ef6ab7eb5e007d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6afeb1878aeb488389ef6ab7eb5e007d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6afeb1878aeb488389ef6ab7eb5e007d" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 39745 } } }
W20260618 23:24:36.493501 13865 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6afeb1878aeb488389ef6ab7eb5e007d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260618 23:24:36.493774 13851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6afeb1878aeb488389ef6ab7eb5e007d [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:36.494082 13853 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6afeb1878aeb488389ef6ab7eb5e007d [sys.catalog]: This master's current role is: LEADER
W20260618 23:24:36.494244 13865 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:24:36.494333 13868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:24:36.494493 13868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:24:36.496006 13868 catalog_manager.cc:1347] Generated new cluster ID: 40d7f2f5f465462995c4b5fc588dd689
I20260618 23:24:36.496061 13868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:24:36.516527 13868 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:24:36.517033 13868 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:24:36.523862 13868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6afeb1878aeb488389ef6ab7eb5e007d: Generated new TSK 0
I20260618 23:24:36.524055 13868 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:24:36.595216 13849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:36.528870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40412 (local address 127.8.39.190:39745)
0618 23:24:36.528928 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:36.528930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:36.528954 (+ 24us) server_negotiation.cc:407] Connection header received
0618 23:24:36.528982 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:36.528983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:36.528992 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:36.529056 (+ 64us) server_negotiation.cc:226] Negotiated authn=JWT
0618 23:24:36.529255 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:36.529741 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:36.530884 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:36.531105 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:36.531164 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:36.531486 (+ 322us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0618 23:24:36.594998 (+ 63512us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0618 23:24:36.595159 (+ 161us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40412: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":35}
W20260618 23:24:36.595314 13849 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40412: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260618 23:24:36.595388 13875 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:36.528712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:39745 (local address 127.0.0.1:40412)
0618 23:24:36.528853 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:36.528865 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:36.528913 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:36.529079 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:36.529080 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:36.529082 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0618 23:24:36.529176 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:36.529178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:36.529825 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:36.529826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:36.530785 (+ 959us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:36.530787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:36.531198 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:36.531199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:36.531258 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:36.531264 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0618 23:24:36.595153 (+ 63889us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0618 23:24:36.595292 (+ 139us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.39.190:39745: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":82,"threads_started":1}
I20260618 23:24:36.595726 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 13779
2026-06-18T23:24:36Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (250 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (87 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (87 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:24:36 dist-test-slave-47qt krb5kdc[13887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:24:36 dist-test-slave-47qt krb5kdc[13887](info): set up 2 sockets
Jun 18 23:24:36 dist-test-slave-47qt krb5kdc[13887](info): commencing operation
krb5kdc: starting...
W20260618 23:24:37.817502 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:24:37 dist-test-slave-47qt krb5kdc[13887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:24:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:24:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.39.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
I20260618 23:24:37.936682 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:38077
--webserver_interface=127.8.39.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:38077
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:38.016350 13900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:38.016556 13900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:38.016578 13900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:38.017769 13900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:24:38.017814 13900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:38.017828 13900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:38.017841 13900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:24:38.017853 13900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:24:38.019295 13900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:38077
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:38077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:38.019536 13900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:38.019739 13900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:38.024490 13908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:38.024506 13906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:38.024637 13905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:38.024776 13900 server_base.cc:1034] running on GCE node
I20260618 23:24:38.025048 13900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:38.025349 13900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:38.026502 13900 hybrid_clock.cc:648] HybridClock initialized: now 1781825078026489 us; error 30 us; skew 500 ppm
Jun 18 23:24:38 dist-test-slave-47qt krb5kdc[13887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825078, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:38.029177 13900 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:24:38.029623 13900 webserver.cc:460] Webserver started at http://127.8.39.190:43679/ using document root <none> and password file <none>
I20260618 23:24:38.029855 13900 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:38.029899 13900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:38.030040 13900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:38.031052 13900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "fcc4c78dc83e438f9a5b90956d7ef907"
format_stamp: "Formatted at 2026-06-18 23:24:38 on dist-test-slave-47qt"
server_key: "eb8818edca5415c174ed7537e711d26d"
server_key_iv: "9de48bed3796207f0bcf5e86c947c504"
server_key_version: "encryptionkey@0"
I20260618 23:24:38.031401 13900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "fcc4c78dc83e438f9a5b90956d7ef907"
format_stamp: "Formatted at 2026-06-18 23:24:38 on dist-test-slave-47qt"
server_key: "eb8818edca5415c174ed7537e711d26d"
server_key_iv: "9de48bed3796207f0bcf5e86c947c504"
server_key_version: "encryptionkey@0"
I20260618 23:24:38.032905 13900 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:24:38.033790 13915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:38.034017 13900 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:24:38.034092 13900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "fcc4c78dc83e438f9a5b90956d7ef907"
format_stamp: "Formatted at 2026-06-18 23:24:38 on dist-test-slave-47qt"
server_key: "eb8818edca5415c174ed7537e711d26d"
server_key_iv: "9de48bed3796207f0bcf5e86c947c504"
server_key_version: "encryptionkey@0"
I20260618 23:24:38.034175 13900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:38.051545 13900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:38.053634 13900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:38.053763 13900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:38.057585 13900 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:38077
I20260618 23:24:38.057691 13967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:38077 every 8 connection(s)
I20260618 23:24:38.057984 13900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:24:38.058785 13968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:38.060927 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 13900
I20260618 23:24:38.061033 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:24:38.061151 13968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcc4c78dc83e438f9a5b90956d7ef907: Bootstrap starting.
I20260618 23:24:38.061273 8350 external_mini_cluster.cc:1428] Setting key c1a232c7e07e3feb5ec75f1dcd3bf847
I20260618 23:24:38.061807 13968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcc4c78dc83e438f9a5b90956d7ef907: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:38.062137 13968 log.cc:824] T 00000000000000000000000000000000 P fcc4c78dc83e438f9a5b90956d7ef907: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:38.062865 13968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcc4c78dc83e438f9a5b90956d7ef907: No bootstrap required, opened a new log
I20260618 23:24:38.065449 13968 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcc4c78dc83e438f9a5b90956d7ef907 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcc4c78dc83e438f9a5b90956d7ef907" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 38077 } }
I20260618 23:24:38.065584 13968 raft_consensus.cc:374] T 00000000000000000000000000000000 P fcc4c78dc83e438f9a5b90956d7ef907 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:38.065608 13968 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcc4c78dc83e438f9a5b90956d7ef907 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcc4c78dc83e438f9a5b90956d7ef907, State: Initialized, Role: FOLLOWER
I20260618 23:24:38.065753 13968 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcc4c78dc83e438f9a5b90956d7ef907 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcc4c78dc83e438f9a5b90956d7ef907" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 38077 } }
I20260618 23:24:38.065819 13968 raft_consensus.cc:388] T 00000000000000000000000000000000 P fcc4c78dc83e438f9a5b90956d7ef907 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:24:38.065871 13968 raft_consensus.cc:482] T 00000000000000000000000000000000 P fcc4c78dc83e438f9a5b90956d7ef907 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:24:38.065943 13968 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcc4c78dc83e438f9a5b90956d7ef907 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:38.066514 13968 raft_consensus.cc:504] T 00000000000000000000000000000000 P fcc4c78dc83e438f9a5b90956d7ef907 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcc4c78dc83e438f9a5b90956d7ef907" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 38077 } }
I20260618 23:24:38.066634 13968 leader_election.cc:302] T 00000000000000000000000000000000 P fcc4c78dc83e438f9a5b90956d7ef907 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fcc4c78dc83e438f9a5b90956d7ef907; no voters:
I20260618 23:24:38.066802 13968 leader_election.cc:288] T 00000000000000000000000000000000 P fcc4c78dc83e438f9a5b90956d7ef907 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:24:38.067006 13974 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fcc4c78dc83e438f9a5b90956d7ef907 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:38.067044 13968 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcc4c78dc83e438f9a5b90956d7ef907 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:24:38.068149 13974 raft_consensus.cc:686] T 00000000000000000000000000000000 P fcc4c78dc83e438f9a5b90956d7ef907 [term 1 LEADER]: Becoming Leader. State: Replica: fcc4c78dc83e438f9a5b90956d7ef907, State: Running, Role: LEADER
I20260618 23:24:38.068264 13974 consensus_queue.cc:237] T 00000000000000000000000000000000 P fcc4c78dc83e438f9a5b90956d7ef907 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcc4c78dc83e438f9a5b90956d7ef907" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 38077 } }
Jun 18 23:24:38 dist-test-slave-47qt krb5kdc[13887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:38.068850 13973 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcc4c78dc83e438f9a5b90956d7ef907 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fcc4c78dc83e438f9a5b90956d7ef907" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcc4c78dc83e438f9a5b90956d7ef907" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 38077 } } }
I20260618 23:24:38.068925 13973 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcc4c78dc83e438f9a5b90956d7ef907 [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:38.069238 13983 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcc4c78dc83e438f9a5b90956d7ef907 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fcc4c78dc83e438f9a5b90956d7ef907. Latest consensus state: current_term: 1 leader_uuid: "fcc4c78dc83e438f9a5b90956d7ef907" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fcc4c78dc83e438f9a5b90956d7ef907" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 38077 } } }
I20260618 23:24:38.069331 13983 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcc4c78dc83e438f9a5b90956d7ef907 [sys.catalog]: This master's current role is: LEADER
W20260618 23:24:38.069238 13987 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fcc4c78dc83e438f9a5b90956d7ef907: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 23:24:38.069485 13987 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:24:38.069821 13989 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:24:38.070001 13989 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:24:38.071571 13989 catalog_manager.cc:1347] Generated new cluster ID: 8f753c27905149b2bb65f17eda10d05b
I20260618 23:24:38.071622 13989 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:24:38.073391 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:38.061771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54876 (local address 127.8.39.190:38077)
0618 23:24:38.061991 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:38.062002 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:38.062032 (+ 30us) server_negotiation.cc:407] Connection header received
0618 23:24:38.062579 (+ 547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:38.062587 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:38.062645 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:38.062798 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:38.063593 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:38.065085 (+ 1492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:38.066060 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:38.066467 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:38.066606 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:38.069516 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:38.069520 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:38.069520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:38.069532 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:38.071630 (+ 2098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:38.072101 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:38.072103 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:38.072104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:38.072199 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:38.072542 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:38.072543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:38.072543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:38.072780 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:38.072912 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:38.073024 (+ 112us) server_negotiation.cc:299] Negotiation successful
0618 23:24:38.073189 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":72,"threads_started":1}
I20260618 23:24:38.082963 13989 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:24:38.083709 13989 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:24:38.090122 13989 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fcc4c78dc83e438f9a5b90956d7ef907: Generated new TSK 0
I20260618 23:24:38.090310 13989 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.39.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
I20260618 23:24:38.120550 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.129:0
--local_ip_for_outbound_sockets=127.8.39.129
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:38077
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:38.203035 13994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:38.203239 13994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:38.203263 13994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:38.204424 13994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:38.204469 13994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:38.204495 13994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.129
I20260618 23:24:38.205977 13994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:38077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.129
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:38.206216 13994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:38.206466 13994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:38.211738 13999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:38.211738 14000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:38.211740 14002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:38.211998 13994 server_base.cc:1034] running on GCE node
I20260618 23:24:38.212203 13994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:38.212445 13994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:38.213609 13994 hybrid_clock.cc:648] HybridClock initialized: now 1781825078213593 us; error 24 us; skew 500 ppm
Jun 18 23:24:38 dist-test-slave-47qt krb5kdc[13887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825078, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:38.216231 13994 init.cc:377] Logged in from keytab as kudu/127.8.39.129@KRBTEST.COM (short username kudu)
I20260618 23:24:38.216991 13994 webserver.cc:460] Webserver started at http://127.8.39.129:38595/ using document root <none> and password file <none>
I20260618 23:24:38.217280 13994 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:38.217352 13994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:38.217509 13994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:38.218576 13994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-0/data/instance:
uuid: "72c71ec155174482b9ea5493a58c5880"
format_stamp: "Formatted at 2026-06-18 23:24:38 on dist-test-slave-47qt"
server_key: "9f753776e2afdb9e47d691af9f32c005"
server_key_iv: "64b0e8710d49c60fd47d189d77542803"
server_key_version: "encryptionkey@0"
I20260618 23:24:38.218948 13994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance:
uuid: "72c71ec155174482b9ea5493a58c5880"
format_stamp: "Formatted at 2026-06-18 23:24:38 on dist-test-slave-47qt"
server_key: "9f753776e2afdb9e47d691af9f32c005"
server_key_iv: "64b0e8710d49c60fd47d189d77542803"
server_key_version: "encryptionkey@0"
I20260618 23:24:38.220293 13994 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:24:38.221155 14009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:38.221367 13994 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:38.221467 13994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-0/wal
uuid: "72c71ec155174482b9ea5493a58c5880"
format_stamp: "Formatted at 2026-06-18 23:24:38 on dist-test-slave-47qt"
server_key: "9f753776e2afdb9e47d691af9f32c005"
server_key_iv: "64b0e8710d49c60fd47d189d77542803"
server_key_version: "encryptionkey@0"
I20260618 23:24:38.221539 13994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:38.250452 13994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:38.252573 13994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:38.252735 13994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:38.253021 13994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:38.253398 13994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:38.253461 13994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:38.253508 13994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:38.253537 13994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:38.259773 13994 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.129:41085
I20260618 23:24:38.259819 14122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.129:41085 every 8 connection(s)
I20260618 23:24:38.260175 13994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
Jun 18 23:24:38 dist-test-slave-47qt krb5kdc[13887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825078, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:38.266721 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 13994
I20260618 23:24:38.266878 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance
I20260618 23:24:38.267129 8350 external_mini_cluster.cc:1428] Setting key b55f1d5cc885f1b46dfcbb85b518ea2f
I20260618 23:24:38.270311 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:38.261112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:49687 (local address 127.8.39.190:38077)
0618 23:24:38.261197 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:38.261199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:38.261345 (+ 146us) server_negotiation.cc:407] Connection header received
0618 23:24:38.262263 (+ 918us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:38.262264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:38.262274 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:38.262347 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:38.263031 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:38.263764 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:38.264785 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:38.264985 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:38.265042 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:38.267333 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:38.267334 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:38.267335 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:38.267342 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:38.268911 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:38.269386 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:38.269387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:38.269387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:38.269446 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:38.269695 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:38.269696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:38.269696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:38.269805 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:38.269877 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:38.270142 (+ 265us) server_negotiation.cc:299] Negotiation successful
0618 23:24:38.270224 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 23:24:38.270505 14125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:38.261007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:38077 (local address 127.8.39.129:49687)
0618 23:24:38.261218 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:38.261253 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:38.262141 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:38.262447 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:38.262451 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:38.262706 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:38.262916 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:38.262920 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:38.263890 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:38.263891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:38.264692 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:38.264694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:38.265060 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:38.265061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:38.265174 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:38.265722 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:38.265729 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:38.267230 (+ 1501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:38.268998 (+ 1768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:38.269004 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:38.269006 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:38.269300 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:38.269508 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:38.269509 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:38.269509 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:38.269619 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:38.269869 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:38.269874 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:38.270010 (+ 136us) client_negotiation.cc:769] Sending connection context
0618 23:24:38.270108 (+ 98us) client_negotiation.cc:240] Negotiation successful
0618 23:24:38.270257 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":75,"threads_started":1}
I20260618 23:24:38.270931 14123 heartbeater.cc:347] Connected to a master server at 127.8.39.190:38077
I20260618 23:24:38.271032 14123 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:38.271245 14123 heartbeater.cc:511] Master 127.8.39.190:38077 requested a full tablet report, sending...
I20260618 23:24:38.271756 13932 ts_manager.cc:194] Registered new tserver with Master: 72c71ec155174482b9ea5493a58c5880 (127.8.39.129:41085)
I20260618 23:24:38.272380 13932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.129@KRBTEST.COM'} at 127.8.39.129:49687
WARNING: no policy specified for kudu/127.8.39.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
I20260618 23:24:38.293326 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.130:0
--local_ip_for_outbound_sockets=127.8.39.130
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:38077
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:38.376936 14128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:38.377213 14128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:38.377251 14128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:38.378446 14128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:38.378485 14128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:38.378511 14128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.130
I20260618 23:24:38.379971 14128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:38077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14128
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.130
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:38.380218 14128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:38.380458 14128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:38.385564 14136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:38.385619 14133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:38.385576 14134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:38.385768 14128 server_base.cc:1034] running on GCE node
I20260618 23:24:38.386021 14128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:38.386214 14128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:38.387369 14128 hybrid_clock.cc:648] HybridClock initialized: now 1781825078387348 us; error 39 us; skew 500 ppm
Jun 18 23:24:38 dist-test-slave-47qt krb5kdc[13887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825078, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:38.390012 14128 init.cc:377] Logged in from keytab as kudu/127.8.39.130@KRBTEST.COM (short username kudu)
I20260618 23:24:38.390478 14128 webserver.cc:460] Webserver started at http://127.8.39.130:33991/ using document root <none> and password file <none>
I20260618 23:24:38.390703 14128 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:38.390749 14128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:38.390906 14128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:38.391956 14128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-1/data/instance:
uuid: "aea3684bea594c2d98ed59c08dbd74b3"
format_stamp: "Formatted at 2026-06-18 23:24:38 on dist-test-slave-47qt"
server_key: "c8eb2d85bbd542cc74e31b4333da07c6"
server_key_iv: "f5424f1d15b4e4eb80ace87567e5aa4d"
server_key_version: "encryptionkey@0"
I20260618 23:24:38.392297 14128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance:
uuid: "aea3684bea594c2d98ed59c08dbd74b3"
format_stamp: "Formatted at 2026-06-18 23:24:38 on dist-test-slave-47qt"
server_key: "c8eb2d85bbd542cc74e31b4333da07c6"
server_key_iv: "f5424f1d15b4e4eb80ace87567e5aa4d"
server_key_version: "encryptionkey@0"
I20260618 23:24:38.393868 14128 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260618 23:24:38.394681 14143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:38.394905 14128 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:38.395007 14128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-1/wal
uuid: "aea3684bea594c2d98ed59c08dbd74b3"
format_stamp: "Formatted at 2026-06-18 23:24:38 on dist-test-slave-47qt"
server_key: "c8eb2d85bbd542cc74e31b4333da07c6"
server_key_iv: "f5424f1d15b4e4eb80ace87567e5aa4d"
server_key_version: "encryptionkey@0"
I20260618 23:24:38.395084 14128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:38.404531 14128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:38.406823 14128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:38.406993 14128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:38.407222 14128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:38.407518 14128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:38.407575 14128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:38.407610 14128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:38.407637 14128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:38.413673 14128 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.130:38923
I20260618 23:24:38.413712 14256 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.130:38923 every 8 connection(s)
I20260618 23:24:38.414037 14128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
I20260618 23:24:38.418463 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 14128
I20260618 23:24:38.418555 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance
I20260618 23:24:38.418840 8350 external_mini_cluster.cc:1428] Setting key e2c107af91ff68e65ec9316919f02dec
Jun 18 23:24:38 dist-test-slave-47qt krb5kdc[13887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825078, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:38.425055 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:38.415259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:55719 (local address 127.8.39.190:38077)
0618 23:24:38.415317 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:38.415320 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:38.415572 (+ 252us) server_negotiation.cc:407] Connection header received
0618 23:24:38.416551 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:38.416552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:38.416564 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:38.416616 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:38.417282 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:38.417769 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:38.418869 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:38.419192 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:38.419256 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:38.421979 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:38.421981 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:38.421981 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:38.421990 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:38.423447 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:38.423883 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:38.423884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:38.423885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:38.423944 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:38.424230 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:38.424231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:38.424231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:38.424417 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:38.424492 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:38.424757 (+ 265us) server_negotiation.cc:299] Negotiation successful
0618 23:24:38.424938 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260618 23:24:38.425216 14259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:38.415259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:38077 (local address 127.8.39.130:55719)
0618 23:24:38.415446 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:38.415481 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:38.416415 (+ 934us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:38.416689 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:38.416693 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:38.416947 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:38.417159 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:38.417163 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:38.417871 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:38.417872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:38.418773 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:38.418776 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:38.419282 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:38.419283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:38.419452 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:38.420003 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:38.420013 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:38.421881 (+ 1868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:38.423553 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:38.423560 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:38.423563 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:38.423791 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:38.424011 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:38.424012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:38.424012 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:38.424167 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:38.424483 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:38.424489 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:38.424628 (+ 139us) client_negotiation.cc:769] Sending connection context
0618 23:24:38.424746 (+ 118us) client_negotiation.cc:240] Negotiation successful
0618 23:24:38.424924 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":90,"threads_started":1}
I20260618 23:24:38.425645 14257 heartbeater.cc:347] Connected to a master server at 127.8.39.190:38077
I20260618 23:24:38.425752 14257 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:38.425959 14257 heartbeater.cc:511] Master 127.8.39.190:38077 requested a full tablet report, sending...
I20260618 23:24:38.426335 13932 ts_manager.cc:194] Registered new tserver with Master: aea3684bea594c2d98ed59c08dbd74b3 (127.8.39.130:38923)
I20260618 23:24:38.426710 13932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.130@KRBTEST.COM'} at 127.8.39.130:55719
WARNING: no policy specified for kudu/127.8.39.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
I20260618 23:24:38.447881 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.131:0
--local_ip_for_outbound_sockets=127.8.39.131
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:38077
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:44309
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:38.528931 14262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:38.529155 14262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:38.529179 14262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:38.530251 14262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:38.530290 14262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:38.530316 14262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.131
I20260618 23:24:38.531687 14262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:44309
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:38077
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14262
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.131
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:38.531860 14262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:38.532110 14262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:38.538020 14270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:38.538084 14262 server_base.cc:1034] running on GCE node
W20260618 23:24:38.538036 14268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:38.538021 14267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:38.538676 14262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:38.538993 14262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:38.540170 14262 hybrid_clock.cc:648] HybridClock initialized: now 1781825078540157 us; error 49 us; skew 500 ppm
Jun 18 23:24:38 dist-test-slave-47qt krb5kdc[13887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825078, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:38.542779 14262 init.cc:377] Logged in from keytab as kudu/127.8.39.131@KRBTEST.COM (short username kudu)
I20260618 23:24:38.543282 14262 webserver.cc:460] Webserver started at http://127.8.39.131:38503/ using document root <none> and password file <none>
I20260618 23:24:38.543538 14262 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:38.543620 14262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:38.543766 14262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:38.544763 14262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-2/data/instance:
uuid: "8d86fc57133f49478e0e6b385118e80f"
format_stamp: "Formatted at 2026-06-18 23:24:38 on dist-test-slave-47qt"
server_key: "97fff59f90ede6bd8b51980b76d7b19d"
server_key_iv: "37da5cc72bdc27f76741d51fa27565b4"
server_key_version: "encryptionkey@0"
I20260618 23:24:38.545171 14262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance:
uuid: "8d86fc57133f49478e0e6b385118e80f"
format_stamp: "Formatted at 2026-06-18 23:24:38 on dist-test-slave-47qt"
server_key: "97fff59f90ede6bd8b51980b76d7b19d"
server_key_iv: "37da5cc72bdc27f76741d51fa27565b4"
server_key_version: "encryptionkey@0"
I20260618 23:24:38.546515 14262 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:24:38.547341 14277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:38.547523 14262 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:24:38.547627 14262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-2/wal
uuid: "8d86fc57133f49478e0e6b385118e80f"
format_stamp: "Formatted at 2026-06-18 23:24:38 on dist-test-slave-47qt"
server_key: "97fff59f90ede6bd8b51980b76d7b19d"
server_key_iv: "37da5cc72bdc27f76741d51fa27565b4"
server_key_version: "encryptionkey@0"
I20260618 23:24:38.547715 14262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:38.563839 14262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:38.565793 14262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:38.565943 14262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:38.566201 14262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:38.566509 14262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:38.566562 14262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:38.566603 14262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:38.566627 14262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:38.572778 14262 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.131:37701
I20260618 23:24:38.572825 14390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.131:37701 every 8 connection(s)
I20260618 23:24:38.573161 14262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
Jun 18 23:24:38 dist-test-slave-47qt krb5kdc[13887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825078, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:38.582746 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 14262
I20260618 23:24:38.582841 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance
I20260618 23:24:38.583024 8350 external_mini_cluster.cc:1428] Setting key bdd5dfb5bac7cc97a17bb2215cfd9bb7
I20260618 23:24:38.583105 13971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:38.574022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:56849 (local address 127.8.39.190:38077)
0618 23:24:38.574091 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:38.574093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:38.574294 (+ 201us) server_negotiation.cc:407] Connection header received
0618 23:24:38.575235 (+ 941us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:38.575236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:38.575248 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:38.575318 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:38.575925 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:38.576437 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:38.577690 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:38.577872 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:38.577939 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:38.580071 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:38.580072 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:38.580072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:38.580078 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:38.581460 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:38.581848 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:38.581848 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:38.581849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:38.581898 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:38.582199 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:38.582200 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:38.582200 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:38.582301 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:38.582361 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:38.582916 (+ 555us) server_negotiation.cc:299] Negotiation successful
0618 23:24:38.583010 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260618 23:24:38.583568 14393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:38.574015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:38077 (local address 127.8.39.131:56849)
0618 23:24:38.574171 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:38.574210 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:38.575093 (+ 883us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:38.575385 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:38.575389 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:38.575648 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:38.575827 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:38.575830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:38.576529 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:38.576530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:38.577596 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:38.577598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:38.577976 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:38.577977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:38.578091 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:38.578644 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:38.578653 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:38.579984 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:38.581549 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:38.581555 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:38.581556 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:38.581778 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:38.582002 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:38.582003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:38.582003 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:38.582126 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:38.582460 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:38.582474 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:38.582677 (+ 203us) client_negotiation.cc:769] Sending connection context
0618 23:24:38.582847 (+ 170us) client_negotiation.cc:240] Negotiation successful
0618 23:24:38.583271 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":92,"threads_started":1}
I20260618 23:24:38.584072 14391 heartbeater.cc:347] Connected to a master server at 127.8.39.190:38077
I20260618 23:24:38.584177 14391 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:38.584370 14391 heartbeater.cc:511] Master 127.8.39.190:38077 requested a full tablet report, sending...
I20260618 23:24:38.584739 13932 ts_manager.cc:194] Registered new tserver with Master: 8d86fc57133f49478e0e6b385118e80f (127.8.39.131:37701)
I20260618 23:24:38.585083 13932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.131@KRBTEST.COM'} at 127.8.39.131:56849
I20260618 23:24:38.595471 8350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/test-admin.keytab.
Jun 18 23:24:38 dist-test-slave-47qt krb5kdc[13887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:38.615909 8350 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260618 23:24:38.617892 14398 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:38.616936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44374 (local address 127.8.39.129:41085)
0618 23:24:38.617250 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:38.617256 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:38.617270 (+ 14us) server_negotiation.cc:407] Connection header received
0618 23:24:38.617310 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:38.617313 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:38.617332 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:38.617425 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:38.617790 (+ 365us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44374: BlockingRecv error: recv got EOF from 127.0.0.1:44374 (error 108)
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":202,"threads_started":1}
W20260618 23:24:38.618395 8350 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/krb5cc' of type FILE
Jun 18 23:24:38 dist-test-slave-47qt krb5kdc[13887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:38.619638 8350 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 18 23:24:38 dist-test-slave-47qt krb5kdc[13887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:38.620788 8350 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 18 23:24:38 dist-test-slave-47qt krb5kdc[13887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.129@KRBTEST.COM
I20260618 23:24:38.631637 14398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:38.621506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44386 (local address 127.8.39.129:41085)
0618 23:24:38.621551 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:38.621553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:38.621577 (+ 24us) server_negotiation.cc:407] Connection header received
0618 23:24:38.621654 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:38.621655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:38.621667 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:38.621717 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:38.622248 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:38.623656 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:38.624841 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:38.625375 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:38.625466 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:38.628300 (+ 2834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:38.628310 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:38.628310 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:38.628318 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:38.630237 (+ 1919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:38.630685 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:38.630688 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:38.630688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:38.630784 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:38.631000 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:38.631001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:38.631001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:38.631157 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:38.631304 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:38.631422 (+ 118us) server_negotiation.cc:299] Negotiation successful
0618 23:24:38.631509 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260618 23:24:38.633513 14398 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:38.632880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44400 (local address 127.8.39.129:41085)
0618 23:24:38.632915 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:38.632916 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:38.632959 (+ 43us) server_negotiation.cc:407] Connection header received
0618 23:24:38.633032 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:38.633033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:38.633041 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:38.633083 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:38.633468 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44400: BlockingRecv error: recv got EOF from 127.0.0.1:44400 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260618 23:24:38.633577 8350 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/krb5cc' of type FILE
Jun 18 23:24:38 dist-test-slave-47qt krb5kdc[13887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:38.634989 8350 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 18 23:24:38 dist-test-slave-47qt krb5kdc[13887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:38.636106 8350 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 18 23:24:38 dist-test-slave-47qt krb5kdc[13887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.129@KRBTEST.COM
I20260618 23:24:38.643910 14398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:38.636650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44414 (local address 127.8.39.129:41085)
0618 23:24:38.636688 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:38.636690 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:38.636744 (+ 54us) server_negotiation.cc:407] Connection header received
0618 23:24:38.636829 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:38.636830 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:38.636840 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:38.636894 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:38.637390 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:38.638013 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:38.638705 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:38.638953 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:38.639029 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:38.641370 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:38.641371 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:38.641372 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:38.641378 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:38.642653 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:38.643023 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:38.643024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:38.643025 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:38.643081 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:38.643278 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:38.643279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:38.643279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:38.643485 (+ 206us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:38.643620 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:38.643722 (+ 102us) server_negotiation.cc:299] Negotiation successful
0618 23:24:38.643809 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260618 23:24:38.645686 14398 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:38.644853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44416 (local address 127.8.39.129:41085)
0618 23:24:38.644917 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:38.644920 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:38.645006 (+ 86us) server_negotiation.cc:407] Connection header received
0618 23:24:38.645050 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:38.645051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:38.645062 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:38.645167 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:38.645629 (+ 462us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44416: BlockingRecv error: recv got EOF from 127.0.0.1:44416 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260618 23:24:38.646144 8350 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781825053542692-8350-0/krb5kdc/krb5cc' of type FILE
Jun 18 23:24:38 dist-test-slave-47qt krb5kdc[13887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:38.647105 8350 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 18 23:24:38 dist-test-slave-47qt krb5kdc[13887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:38.648041 8350 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 18 23:24:38 dist-test-slave-47qt krb5kdc[13887](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825078, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.129@KRBTEST.COM
I20260618 23:24:38.655786 14398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:38.648679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44418 (local address 127.8.39.129:41085)
0618 23:24:38.648724 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:38.648726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:38.648734 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:24:38.648792 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:38.648793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:38.648800 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:38.648845 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:38.649243 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:38.649722 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:38.650573 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:38.650751 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:38.650808 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:38.653432 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:38.653433 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:38.653433 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:38.653439 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:38.654837 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:38.655159 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:38.655160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:38.655160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:38.655213 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:38.655409 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:38.655410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:38.655410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:38.655510 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:38.655590 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:38.655675 (+ 85us) server_negotiation.cc:299] Negotiation successful
0618 23:24:38.655729 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260618 23:24:38.656517 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 13994
I20260618 23:24:38.661933 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 14128
I20260618 23:24:38.666335 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 14262
I20260618 23:24:38.670718 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 13900
2026-06-18T23:24:38Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1901 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:24:38 dist-test-slave-47qt krb5kdc[14411](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:24:38 dist-test-slave-47qt krb5kdc[14411](info): set up 2 sockets
Jun 18 23:24:38 dist-test-slave-47qt krb5kdc[14411](info): commencing operation
krb5kdc: starting...
W20260618 23:24:39.733443 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:24:39 dist-test-slave-47qt krb5kdc[14411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:24:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:24:39Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.39.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.39.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.39.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.190.keytab.
Entry for principal oryx/127.8.39.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.190.keytab.
I20260618 23:24:39.855636 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:41963
--webserver_interface=127.8.39.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.190.keytab
--principal=oryx/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:41963
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:39.936407 14424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:39.936594 14424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:39.936617 14424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:39.937702 14424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:24:39.937737 14424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:39.937752 14424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:39.937764 14424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:24:39.937776 14424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:24:39.939122 14424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:41963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.190.keytab
--principal=oryx/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:41963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:39.939299 14424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:39.939456 14424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:39.944128 14432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:39.944137 14429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:39.944156 14430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:39.944319 14424 server_base.cc:1034] running on GCE node
I20260618 23:24:39.944644 14424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:39.944912 14424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:39.946091 14424 hybrid_clock.cc:648] HybridClock initialized: now 1781825079946051 us; error 55 us; skew 500 ppm
Jun 18 23:24:39 dist-test-slave-47qt krb5kdc[14411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825079, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:39.952517 14424 init.cc:377] Logged in from keytab as oryx/127.8.39.190@KRBTEST.COM (short username oryx)
I20260618 23:24:39.952989 14424 webserver.cc:460] Webserver started at http://127.8.39.190:42959/ using document root <none> and password file <none>
I20260618 23:24:39.953225 14424 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:39.953279 14424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:39.953369 14424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:39.954391 14424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b"
format_stamp: "Formatted at 2026-06-18 23:24:39 on dist-test-slave-47qt"
server_key: "d39d772a5372afe3c0c8b69ffe785428"
server_key_iv: "ebeb33ce5982cd6590b470e16a624b3a"
server_key_version: "encryptionkey@0"
I20260618 23:24:39.954681 14424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b"
format_stamp: "Formatted at 2026-06-18 23:24:39 on dist-test-slave-47qt"
server_key: "d39d772a5372afe3c0c8b69ffe785428"
server_key_iv: "ebeb33ce5982cd6590b470e16a624b3a"
server_key_version: "encryptionkey@0"
I20260618 23:24:39.955958 14424 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:24:39.956734 14439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:39.956964 14424 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:39.957072 14424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b"
format_stamp: "Formatted at 2026-06-18 23:24:39 on dist-test-slave-47qt"
server_key: "d39d772a5372afe3c0c8b69ffe785428"
server_key_iv: "ebeb33ce5982cd6590b470e16a624b3a"
server_key_version: "encryptionkey@0"
I20260618 23:24:39.957180 14424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:39.979460 14424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:39.981997 14424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:39.982175 14424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:39.986066 14424 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:41963
I20260618 23:24:39.986151 14491 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:41963 every 8 connection(s)
I20260618 23:24:39.986478 14424 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:24:39.987424 14492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:39.989691 14492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b: Bootstrap starting.
I20260618 23:24:39.990265 14492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:39.990334 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 14424
I20260618 23:24:39.990464 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:24:39.990612 14492 log.cc:824] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:39.990664 8350 external_mini_cluster.cc:1428] Setting key f9b75d00795885c9eae29cb5d4527e02
I20260618 23:24:39.991449 14492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b: No bootstrap required, opened a new log
I20260618 23:24:39.993861 14492 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } }
I20260618 23:24:39.994046 14492 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:39.994086 14492 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3074d2dc0694e5fa8b0bfa6b3ff985b, State: Initialized, Role: FOLLOWER
I20260618 23:24:39.994208 14492 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } }
I20260618 23:24:39.994308 14492 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:24:39.994338 14492 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:24:39.994383 14492 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:39.995025 14492 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } }
I20260618 23:24:39.995143 14492 leader_election.cc:302] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3074d2dc0694e5fa8b0bfa6b3ff985b; no voters:
I20260618 23:24:39.995347 14492 leader_election.cc:288] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:24:39.995366 14497 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:39.995474 14497 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [term 1 LEADER]: Becoming Leader. State: Replica: a3074d2dc0694e5fa8b0bfa6b3ff985b, State: Running, Role: LEADER
I20260618 23:24:39.995625 14492 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:24:39.995620 14497 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } }
I20260618 23:24:39.996388 14498 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } }
I20260618 23:24:39.996479 14498 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:39.996799 14499 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3074d2dc0694e5fa8b0bfa6b3ff985b. Latest consensus state: current_term: 1 leader_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } }
I20260618 23:24:39.996870 14499 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:39.997104 14506 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 18 23:24:39 dist-test-slave-47qt krb5kdc[14411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM
I20260618 23:24:39.997830 14506 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:24:39.999308 14506 catalog_manager.cc:1347] Generated new cluster ID: 05a830b28cdd4f759c72bf6b6c1a803a
I20260618 23:24:39.999356 14506 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:24:40.003335 14506 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:24:40.003984 14506 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:24:40.005684 14496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:39.991176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46824 (local address 127.8.39.190:41963)
0618 23:24:39.991347 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:39.991356 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:39.991388 (+ 32us) server_negotiation.cc:407] Connection header received
0618 23:24:39.991846 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:39.991852 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:39.991897 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:39.992004 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:39.992777 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:39.993838 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:39.994864 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:39.995361 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:39.995761 (+ 400us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:39.998253 (+ 2492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:39.998257 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:39.998257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:39.998269 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:40.003917 (+ 5648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:40.004350 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:40.004353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:40.004353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:40.004451 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:40.004695 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:40.004696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:40.004696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:40.004916 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:40.005178 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:40.005249 (+ 71us) server_negotiation.cc:299] Negotiation successful
0618 23:24:40.005521 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260618 23:24:40.011430 14506 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b: Generated new TSK 0
I20260618 23:24:40.011628 14506 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.39.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.39.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.39.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.129.keytab.
Entry for principal oryx/127.8.39.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.129.keytab.
I20260618 23:24:40.039930 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.129:0
--local_ip_for_outbound_sockets=127.8.39.129
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:41963
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.129.keytab
--principal=oryx/127.8.39.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:40.157440 14518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:40.157711 14518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:40.157748 14518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:40.159315 14518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:40.159375 14518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:40.159415 14518 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.129
I20260618 23:24:40.161451 14518 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.129.keytab
--principal=oryx/127.8.39.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:41963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.129
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:40.161705 14518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:40.161965 14518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:40.169032 14523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:40.169044 14526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:40.169406 14518 server_base.cc:1034] running on GCE node
W20260618 23:24:40.169049 14524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:40.169687 14518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:40.169937 14518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:40.171136 14518 hybrid_clock.cc:648] HybridClock initialized: now 1781825080171118 us; error 30 us; skew 500 ppm
Jun 18 23:24:40 dist-test-slave-47qt krb5kdc[14411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825080, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:40.174263 14518 init.cc:377] Logged in from keytab as oryx/127.8.39.129@KRBTEST.COM (short username oryx)
I20260618 23:24:40.174732 14518 webserver.cc:460] Webserver started at http://127.8.39.129:35711/ using document root <none> and password file <none>
I20260618 23:24:40.174989 14518 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:40.175057 14518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:40.175176 14518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:40.176234 14518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-0/data/instance:
uuid: "b46d97a84c2b4d88b79434d5e2bf1a16"
format_stamp: "Formatted at 2026-06-18 23:24:40 on dist-test-slave-47qt"
server_key: "693362169c722a21453370b52e8ce402"
server_key_iv: "2217625ad8d9402d55b5c43155847c8e"
server_key_version: "encryptionkey@0"
I20260618 23:24:40.176590 14518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance:
uuid: "b46d97a84c2b4d88b79434d5e2bf1a16"
format_stamp: "Formatted at 2026-06-18 23:24:40 on dist-test-slave-47qt"
server_key: "693362169c722a21453370b52e8ce402"
server_key_iv: "2217625ad8d9402d55b5c43155847c8e"
server_key_version: "encryptionkey@0"
I20260618 23:24:40.178104 14518 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:24:40.178943 14533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:40.179178 14518 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:24:40.179278 14518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-0/wal
uuid: "b46d97a84c2b4d88b79434d5e2bf1a16"
format_stamp: "Formatted at 2026-06-18 23:24:40 on dist-test-slave-47qt"
server_key: "693362169c722a21453370b52e8ce402"
server_key_iv: "2217625ad8d9402d55b5c43155847c8e"
server_key_version: "encryptionkey@0"
I20260618 23:24:40.179354 14518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:40.190145 14518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:40.192198 14518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:40.192373 14518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:40.192587 14518 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:40.192880 14518 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:40.192934 14518 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:40.192974 14518 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:40.193002 14518 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:40.199352 14518 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.129:45209
I20260618 23:24:40.199393 14646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.129:45209 every 8 connection(s)
I20260618 23:24:40.199708 14518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
I20260618 23:24:40.205744 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 14518
I20260618 23:24:40.205829 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance
I20260618 23:24:40.206027 8350 external_mini_cluster.cc:1428] Setting key 4319483cb658000b6f195a9f04a6ce28
Jun 18 23:24:40 dist-test-slave-47qt krb5kdc[14411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825080, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM
I20260618 23:24:40.210186 14649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:40.200400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.8.39.129:56215)
0618 23:24:40.200589 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:40.200621 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:40.201569 (+ 948us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:40.201826 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:40.201829 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:40.202062 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:40.202244 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:40.202247 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.202919 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.202920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:40.203693 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:40.203694 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.204258 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.204259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:40.204371 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:40.204861 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:40.204869 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:40.206475 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:40.208319 (+ 1844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:40.208325 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:40.208327 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:40.208532 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:40.208927 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:40.208927 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:40.208928 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:40.209041 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:40.209642 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:40.209647 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:40.209762 (+ 115us) client_negotiation.cc:769] Sending connection context
0618 23:24:40.209858 (+ 96us) client_negotiation.cc:240] Negotiation successful
0618 23:24:40.209980 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":122,"threads_started":1}
I20260618 23:24:40.210227 14496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:40.200439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:56215 (local address 127.8.39.190:41963)
0618 23:24:40.200515 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:40.200518 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:40.200705 (+ 187us) server_negotiation.cc:407] Connection header received
0618 23:24:40.201682 (+ 977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:40.201683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:40.201693 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:40.201756 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:40.202335 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.202829 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.203868 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.204119 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.204227 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:40.206579 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:40.206581 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:40.206581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:40.206589 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:40.208207 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:40.208604 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:40.208605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:40.208606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:40.208673 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:40.209268 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:40.209271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:40.209271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:40.209539 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:40.209627 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:40.209905 (+ 278us) server_negotiation.cc:299] Negotiation successful
0618 23:24:40.210020 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260618 23:24:40.210758 14647 heartbeater.cc:347] Connected to a master server at 127.8.39.190:41963
I20260618 23:24:40.210873 14647 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:40.211063 14647 heartbeater.cc:511] Master 127.8.39.190:41963 requested a full tablet report, sending...
I20260618 23:24:40.211615 14456 ts_manager.cc:194] Registered new tserver with Master: b46d97a84c2b4d88b79434d5e2bf1a16 (127.8.39.129:45209)
I20260618 23:24:40.212589 14456 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.39.129@KRBTEST.COM'} at 127.8.39.129:56215
WARNING: no policy specified for oryx/127.8.39.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.39.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.39.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.130.keytab.
Entry for principal oryx/127.8.39.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.130.keytab.
I20260618 23:24:40.233541 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.130:0
--local_ip_for_outbound_sockets=127.8.39.130
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:41963
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.130.keytab
--principal=oryx/127.8.39.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:40.312961 14652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:40.313231 14652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:40.313273 14652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:40.314406 14652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:40.314474 14652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:40.314517 14652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.130
I20260618 23:24:40.315901 14652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.130.keytab
--principal=oryx/127.8.39.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:41963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.130
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:40.316182 14652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:40.316411 14652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 23:24:40.321499 14652 server_base.cc:1034] running on GCE node
W20260618 23:24:40.321467 14658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:40.321460 14657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:40.321503 14660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:40.321920 14652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:40.322165 14652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:40.323338 14652 hybrid_clock.cc:648] HybridClock initialized: now 1781825080323323 us; error 26 us; skew 500 ppm
Jun 18 23:24:40 dist-test-slave-47qt krb5kdc[14411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825080, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:40.325831 14652 init.cc:377] Logged in from keytab as oryx/127.8.39.130@KRBTEST.COM (short username oryx)
I20260618 23:24:40.326287 14652 webserver.cc:460] Webserver started at http://127.8.39.130:44477/ using document root <none> and password file <none>
I20260618 23:24:40.326460 14652 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:40.326499 14652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:40.326583 14652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:40.327533 14652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-1/data/instance:
uuid: "a51df27f83cb4e5cab3bcb7bf4d8efa3"
format_stamp: "Formatted at 2026-06-18 23:24:40 on dist-test-slave-47qt"
server_key: "b3bbab56c353eaae033d9aec345aec1b"
server_key_iv: "606c3a84d00884d84b8592a38862aca8"
server_key_version: "encryptionkey@0"
I20260618 23:24:40.327816 14652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance:
uuid: "a51df27f83cb4e5cab3bcb7bf4d8efa3"
format_stamp: "Formatted at 2026-06-18 23:24:40 on dist-test-slave-47qt"
server_key: "b3bbab56c353eaae033d9aec345aec1b"
server_key_iv: "606c3a84d00884d84b8592a38862aca8"
server_key_version: "encryptionkey@0"
I20260618 23:24:40.329023 14652 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260618 23:24:40.329766 14667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:40.329932 14652 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260618 23:24:40.330003 14652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-1/wal
uuid: "a51df27f83cb4e5cab3bcb7bf4d8efa3"
format_stamp: "Formatted at 2026-06-18 23:24:40 on dist-test-slave-47qt"
server_key: "b3bbab56c353eaae033d9aec345aec1b"
server_key_iv: "606c3a84d00884d84b8592a38862aca8"
server_key_version: "encryptionkey@0"
I20260618 23:24:40.330097 14652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:40.346596 14652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:40.348971 14652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:40.349124 14652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:40.349403 14652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:40.349715 14652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:40.349772 14652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:40.349813 14652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:40.349840 14652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:40.356309 14652 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.130:42141
I20260618 23:24:40.356354 14780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.130:42141 every 8 connection(s)
I20260618 23:24:40.356679 14652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
I20260618 23:24:40.358034 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 14652
I20260618 23:24:40.358141 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance
I20260618 23:24:40.358346 8350 external_mini_cluster.cc:1428] Setting key 9991817ce979c0842917b0c61e70c631
Jun 18 23:24:40 dist-test-slave-47qt krb5kdc[14411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825080, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.8.39.131@KRBTEST.COM; defaulting to no policy
I20260618 23:24:40.368037 14496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:40.357409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:50357 (local address 127.8.39.190:41963)
0618 23:24:40.357470 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:40.357488 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:40.357660 (+ 172us) server_negotiation.cc:407] Connection header received
0618 23:24:40.358177 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:40.358178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:40.358190 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:40.358274 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:40.359404 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.360215 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.361813 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.362135 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.362216 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:40.364769 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:40.364770 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:40.364770 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:40.364777 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:40.366592 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:40.367088 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:40.367089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:40.367090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:40.367144 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:40.367485 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:40.367486 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:40.367486 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:40.367590 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:40.367652 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:40.367911 (+ 259us) server_negotiation.cc:299] Negotiation successful
0618 23:24:40.367972 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260618 23:24:40.368386 14783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:40.357375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.8.39.130:50357)
0618 23:24:40.357540 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:40.357581 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:40.358056 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:40.358336 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:40.358339 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:40.359121 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:40.359337 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:40.359341 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.360323 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.360324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:40.361713 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:40.361716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.362226 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.362227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:40.362336 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:40.362886 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:40.362894 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:40.364650 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:40.366701 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:40.366708 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:40.366710 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:40.366992 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:40.367271 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:40.367272 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:40.367272 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:40.367413 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:40.367669 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:40.367675 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:40.367793 (+ 118us) client_negotiation.cc:769] Sending connection context
0618 23:24:40.367888 (+ 95us) client_negotiation.cc:240] Negotiation successful
0618 23:24:40.368167 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":87,"threads_started":1}
I20260618 23:24:40.368847 14781 heartbeater.cc:347] Connected to a master server at 127.8.39.190:41963
I20260618 23:24:40.368965 14781 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:40.369196 14781 heartbeater.cc:511] Master 127.8.39.190:41963 requested a full tablet report, sending...
I20260618 23:24:40.369547 14456 ts_manager.cc:194] Registered new tserver with Master: a51df27f83cb4e5cab3bcb7bf4d8efa3 (127.8.39.130:42141)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.39.131@KRBTEST.COM" created.
I20260618 23:24:40.370074 14456 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.39.130@KRBTEST.COM'} at 127.8.39.130:50357
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.39.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.131.keytab.
Entry for principal oryx/127.8.39.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.131.keytab.
I20260618 23:24:40.386313 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.131:0
--local_ip_for_outbound_sockets=127.8.39.131
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:41963
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.131.keytab
--principal=oryx/127.8.39.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:40.468884 14786 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:40.469122 14786 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:40.469187 14786 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:40.470294 14786 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:40.470367 14786 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:40.470412 14786 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.131
I20260618 23:24:40.471866 14786 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.131.keytab
--principal=oryx/127.8.39.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:41963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14786
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.131
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:40.472132 14786 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:40.472369 14786 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:40.477594 14791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:40.477650 14794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:40.477599 14792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:40.477887 14786 server_base.cc:1034] running on GCE node
I20260618 23:24:40.478170 14786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:40.478425 14786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:40.479621 14786 hybrid_clock.cc:648] HybridClock initialized: now 1781825080479592 us; error 49 us; skew 500 ppm
Jun 18 23:24:40 dist-test-slave-47qt krb5kdc[14411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825080, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:40.482349 14786 init.cc:377] Logged in from keytab as oryx/127.8.39.131@KRBTEST.COM (short username oryx)
I20260618 23:24:40.482897 14786 webserver.cc:460] Webserver started at http://127.8.39.131:35577/ using document root <none> and password file <none>
I20260618 23:24:40.483134 14786 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:40.483203 14786 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:40.483336 14786 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:40.484366 14786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-2/data/instance:
uuid: "e4889f468084485892fd900728fd9312"
format_stamp: "Formatted at 2026-06-18 23:24:40 on dist-test-slave-47qt"
server_key: "0ea8f999a5dddebf3b0477a22a6c7faa"
server_key_iv: "492ec62d2845727ab87301ae570d9c8e"
server_key_version: "encryptionkey@0"
I20260618 23:24:40.484740 14786 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance:
uuid: "e4889f468084485892fd900728fd9312"
format_stamp: "Formatted at 2026-06-18 23:24:40 on dist-test-slave-47qt"
server_key: "0ea8f999a5dddebf3b0477a22a6c7faa"
server_key_iv: "492ec62d2845727ab87301ae570d9c8e"
server_key_version: "encryptionkey@0"
I20260618 23:24:40.486708 14786 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260618 23:24:40.487718 14801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:40.487941 14786 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:40.488049 14786 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-2/wal
uuid: "e4889f468084485892fd900728fd9312"
format_stamp: "Formatted at 2026-06-18 23:24:40 on dist-test-slave-47qt"
server_key: "0ea8f999a5dddebf3b0477a22a6c7faa"
server_key_iv: "492ec62d2845727ab87301ae570d9c8e"
server_key_version: "encryptionkey@0"
I20260618 23:24:40.488142 14786 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:40.497184 14786 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:40.499606 14786 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:40.499773 14786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:40.500020 14786 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:40.500331 14786 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:40.500385 14786 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:40.500424 14786 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:40.500451 14786 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:40.506435 14786 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.131:43739
I20260618 23:24:40.506474 14914 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.131:43739 every 8 connection(s)
I20260618 23:24:40.506783 14786 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
I20260618 23:24:40.510802 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 14786
I20260618 23:24:40.510883 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance
I20260618 23:24:40.511104 8350 external_mini_cluster.cc:1428] Setting key 2482d3b38ff7f495112e5d8800465580
Jun 18 23:24:40 dist-test-slave-47qt krb5kdc[14411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825080, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM
I20260618 23:24:40.516029 14496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:40.507447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:38763 (local address 127.8.39.190:41963)
0618 23:24:40.507527 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:40.507529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:40.507633 (+ 104us) server_negotiation.cc:407] Connection header received
0618 23:24:40.508181 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:40.508182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:40.508194 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:40.508250 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:40.509256 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.509758 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.510771 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.511051 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.511131 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:40.513537 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:40.513538 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:40.513538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:40.513544 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:40.514965 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:40.515298 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:40.515299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:40.515299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:40.515345 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:40.515544 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:40.515545 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:40.515545 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:40.515646 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:40.515695 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:40.515912 (+ 217us) server_negotiation.cc:299] Negotiation successful
0618 23:24:40.515969 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 23:24:40.516181 14917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:40.507356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.8.39.131:38763)
0618 23:24:40.507509 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:40.507543 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:40.508053 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:40.508331 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:40.508334 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:40.508895 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:40.509085 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:40.509089 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.509870 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.509872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:40.510674 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:40.510676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.511196 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.511199 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:40.511371 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:40.512013 (+ 642us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:40.512023 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:40.513457 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:40.515035 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:40.515039 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:40.515041 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:40.515233 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:40.515404 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:40.515405 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:40.515405 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:40.515490 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:40.515702 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:40.515706 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:40.515807 (+ 101us) client_negotiation.cc:769] Sending connection context
0618 23:24:40.515886 (+ 79us) client_negotiation.cc:240] Negotiation successful
0618 23:24:40.515993 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":70,"threads_started":1}
I20260618 23:24:40.516626 14915 heartbeater.cc:347] Connected to a master server at 127.8.39.190:41963
I20260618 23:24:40.516713 14915 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:40.516846 14915 heartbeater.cc:511] Master 127.8.39.190:41963 requested a full tablet report, sending...
I20260618 23:24:40.517212 14456 ts_manager.cc:194] Registered new tserver with Master: e4889f468084485892fd900728fd9312 (127.8.39.131:43739)
I20260618 23:24:40.517558 14456 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.39.131@KRBTEST.COM'} at 127.8.39.131:38763
I20260618 23:24:40.523180 8350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.39.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.39.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.39.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.189.keytab.
Entry for principal oryx/127.8.39.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.189.keytab.
I20260618 23:24:40.548444 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.189:44675
--webserver_interface=127.8.39.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.189.keytab
--principal=oryx/127.8.39.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:41963,127.8.39.189:44675
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:40.629400 14920 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:40.629626 14920 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:40.629662 14920 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:40.630786 14920 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:24:40.630854 14920 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:40.630882 14920 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:40.630911 14920 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:24:40.630932 14920 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:24:40.632670 14920 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:41963,127.8.39.189:44675
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.189.keytab
--principal=oryx/127.8.39.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.189:44675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.39.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14920
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
W20260618 23:24:40.632804 14920 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.8.39.190:41963,127.8.39.189:44675'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260618 23:24:40.632959 14920 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:40.633224 14920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:40.637950 14926 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:40.638134 14928 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:40.638176 14925 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:40.638180 14920 server_base.cc:1034] running on GCE node
I20260618 23:24:40.638521 14920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:40.638795 14920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:40.639986 14920 hybrid_clock.cc:648] HybridClock initialized: now 1781825080639973 us; error 30 us; skew 500 ppm
Jun 18 23:24:40 dist-test-slave-47qt krb5kdc[14411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825080, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:40.642604 14920 init.cc:377] Logged in from keytab as oryx/127.8.39.189@KRBTEST.COM (short username oryx)
I20260618 23:24:40.643018 14920 webserver.cc:460] Webserver started at http://127.8.39.189:44123/ using document root <none> and password file <none>
I20260618 23:24:40.643270 14920 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:40.643342 14920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:40.643486 14920 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:40.644490 14920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-1/data/instance:
uuid: "6b7ee65180f14c8cbe67b042b1685a8f"
format_stamp: "Formatted at 2026-06-18 23:24:40 on dist-test-slave-47qt"
server_key: "3ffd758d87cf578047cb02fcf4cbd78c"
server_key_iv: "f80f6639c4bd8e4eb1f0d7050b9baf12"
server_key_version: "encryptionkey@0"
I20260618 23:24:40.644836 14920 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-1/wal/instance:
uuid: "6b7ee65180f14c8cbe67b042b1685a8f"
format_stamp: "Formatted at 2026-06-18 23:24:40 on dist-test-slave-47qt"
server_key: "3ffd758d87cf578047cb02fcf4cbd78c"
server_key_iv: "f80f6639c4bd8e4eb1f0d7050b9baf12"
server_key_version: "encryptionkey@0"
I20260618 23:24:40.646217 14920 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:24:40.647019 14935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:40.647264 14920 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:40.647372 14920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-1/wal
uuid: "6b7ee65180f14c8cbe67b042b1685a8f"
format_stamp: "Formatted at 2026-06-18 23:24:40 on dist-test-slave-47qt"
server_key: "3ffd758d87cf578047cb02fcf4cbd78c"
server_key_iv: "f80f6639c4bd8e4eb1f0d7050b9baf12"
server_key_version: "encryptionkey@0"
I20260618 23:24:40.647456 14920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:40.660527 14920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:40.662889 14920 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:40.663064 14920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:40.666887 14920 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.189:44675
I20260618 23:24:40.666918 14987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.189:44675 every 8 connection(s)
I20260618 23:24:40.667275 14920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-1/data/info.pb
I20260618 23:24:40.668237 14988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:40.669991 14988 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:24:40.673295 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 14920
Jun 18 23:24:40 dist-test-slave-47qt krb5kdc[14411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825080, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.189@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM
Jun 18 23:24:40 dist-test-slave-47qt krb5kdc[14411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.39.189@KRBTEST.COM
I20260618 23:24:40.679658 14496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:40.670715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46828 (local address 127.8.39.190:41963)
0618 23:24:40.670778 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:40.670780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:40.671033 (+ 253us) server_negotiation.cc:407] Connection header received
0618 23:24:40.671518 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:40.671519 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:40.671531 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:40.671579 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:40.672188 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.672700 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.674015 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.674344 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.674424 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:40.676492 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:40.676494 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:40.676494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:40.676503 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:40.678120 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:40.678518 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:40.678519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:40.678519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:40.678579 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:40.678812 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:40.678813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:40.678814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:40.679005 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:40.679094 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:40.679464 (+ 370us) server_negotiation.cc:299] Negotiation successful
0618 23:24:40.679548 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260618 23:24:40.679904 14990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:40.670691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.0.0.1:46828)
0618 23:24:40.670887 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:40.670921 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:40.671407 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:40.671646 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:40.671654 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:40.671906 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:40.672091 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:40.672095 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.672800 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.672801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:40.673925 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:40.673928 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.674427 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.674427 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:40.674536 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:40.675085 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:40.675093 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:40.676399 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:40.678210 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:40.678216 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:40.678218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:40.678437 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:40.678651 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:40.678651 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:40.678652 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:40.678746 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:40.679142 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:40.679148 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:40.679281 (+ 133us) client_negotiation.cc:769] Sending connection context
0618 23:24:40.679427 (+ 146us) client_negotiation.cc:240] Negotiation successful
0618 23:24:40.679655 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":100,"threads_started":1}
I20260618 23:24:40.680680 14988 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:24:40.682900 14992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:40.673803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42278 (local address 127.8.39.189:44675)
0618 23:24:40.673956 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:40.673961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:40.674055 (+ 94us) server_negotiation.cc:407] Connection header received
0618 23:24:40.674073 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:40.674074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:40.674091 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:40.674142 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:40.674792 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.675553 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.676360 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.676658 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.676754 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:40.679526 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:40.679533 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:40.679533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:40.679540 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:40.681698 (+ 2158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:40.682093 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:40.682096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:40.682096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:40.682157 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:40.682337 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:40.682338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:40.682338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:40.682498 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:40.682665 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:40.682719 (+ 54us) server_negotiation.cc:299] Negotiation successful
0618 23:24:40.682795 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
Jun 18 23:24:40 dist-test-slave-47qt krb5kdc[14411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825080, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.189@KRBTEST.COM for oryx/127.8.39.189@KRBTEST.COM
I20260618 23:24:40.688545 14990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:40.680957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:44675 (local address 127.0.0.1:42280)
0618 23:24:40.681162 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:40.681172 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:40.681212 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:40.681353 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:40.681354 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:40.681544 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:40.681588 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:40.681590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.682455 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.682456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:40.683008 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:40.683010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.683493 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.683494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:40.683567 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:40.683877 (+ 310us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:40.683881 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:40.685170 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:40.687142 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:40.687142 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:40.687143 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:40.687302 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:40.687552 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:40.687553 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:40.687553 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:40.687609 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:40.687877 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:40.687878 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:40.687915 (+ 37us) client_negotiation.cc:769] Sending connection context
0618 23:24:40.687986 (+ 71us) client_negotiation.cc:240] Negotiation successful
0618 23:24:40.688260 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260618 23:24:40.688570 14993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:40.681005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42280 (local address 127.8.39.189:44675)
0618 23:24:40.681194 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:40.681196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:40.681205 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:24:40.681261 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:40.681262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:40.681272 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:40.681346 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:40.681658 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.682389 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.683084 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.683411 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.683483 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:40.685283 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:40.685284 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:40.685284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:40.685290 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:40.687007 (+ 1717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:40.687398 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:40.687399 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:40.687400 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:40.687457 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:40.687679 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:40.687680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:40.687680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:40.687791 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:40.687845 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:40.688003 (+ 158us) server_negotiation.cc:299] Negotiation successful
0618 23:24:40.688452 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":288,"server-negotiator.queue_time_us":165,"thread_start_us":87,"threads_started":1}
I20260618 23:24:40.690584 14988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f: Bootstrap starting.
I20260618 23:24:40.691627 14988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:40.691953 14988 log.cc:824] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:40.692703 14988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f: No bootstrap required, opened a new log
I20260618 23:24:40.694180 14988 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } }
I20260618 23:24:40.694309 14988 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:40.694332 14988 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6b7ee65180f14c8cbe67b042b1685a8f, State: Initialized, Role: FOLLOWER
I20260618 23:24:40.694418 14988 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } }
I20260618 23:24:40.694677 14988 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:24:40.694666 14995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } } }
I20260618 23:24:40.694782 14995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [sys.catalog]: This master's current role is: FOLLOWER
W20260618 23:24:40.696561 15006 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 23:24:40.696623 15006 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:24:40.698830 14920 master_runner.cc:428] Detected that this master 6b7ee65180f14c8cbe67b042b1685a8f is joining an existing cluster
I20260618 23:24:40.698874 14920 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.39.189:44675
I20260618 23:24:40.707202 14496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:40.699962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46844 (local address 127.8.39.190:41963)
0618 23:24:40.700120 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:40.700122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:40.700246 (+ 124us) server_negotiation.cc:407] Connection header received
0618 23:24:40.700357 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:40.700358 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:40.700369 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:40.700418 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:40.700837 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.701675 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.702814 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.703124 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.703203 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:40.704381 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:40.704382 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:40.704382 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:40.704389 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:40.705960 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:40.706417 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:40.706418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:40.706418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:40.706474 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:40.706688 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:40.706689 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:40.706690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:40.706804 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:40.706883 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:40.707032 (+ 149us) server_negotiation.cc:299] Negotiation successful
0618 23:24:40.707107 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132}
I20260618 23:24:40.707218 15013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:40.699901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.0.0.1:46844)
0618 23:24:40.700077 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:40.700177 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:40.700274 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:40.700433 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:40.700434 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:40.700644 (+ 210us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:40.700727 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:40.700730 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.701765 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.701766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:40.702721 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:40.702723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.703230 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.703231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:40.703279 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:40.703773 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:40.703778 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:40.704274 (+ 496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:40.706073 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:40.706074 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:40.706074 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:40.706328 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:40.706553 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:40.706554 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:40.706554 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:40.706611 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:40.706873 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:40.706874 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:40.706916 (+ 42us) client_negotiation.cc:769] Sending connection context
0618 23:24:40.707002 (+ 86us) client_negotiation.cc:240] Negotiation successful
0618 23:24:40.707106 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":23,"thread_start_us":85,"threads_started":1}
I20260618 23:24:40.707973 15012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:40.699901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:44675 (local address 127.0.0.1:42288)
0618 23:24:40.700180 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:40.700196 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:40.700271 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:40.700639 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:40.700640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:40.700835 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:40.700898 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:40.700900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.701839 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.701840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:40.702721 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:40.702723 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.703691 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.703691 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:40.703750 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:40.704147 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:40.704152 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:40.704541 (+ 389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:40.706527 (+ 1986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:40.706527 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:40.706528 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:40.706714 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:40.706888 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:40.706889 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:40.706889 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:40.706944 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:40.707212 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:40.707212 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:40.707656 (+ 444us) client_negotiation.cc:769] Sending connection context
0618 23:24:40.707743 (+ 87us) client_negotiation.cc:240] Negotiation successful
0618 23:24:40.707847 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":50,"spinlock_wait_cycles":8448,"thread_start_us":114,"threads_started":1}
I20260618 23:24:40.708040 14993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:40.700104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42288 (local address 127.8.39.189:44675)
0618 23:24:40.700142 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:40.700144 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:40.700255 (+ 111us) server_negotiation.cc:407] Connection header received
0618 23:24:40.700385 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:40.700386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:40.700406 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:40.700560 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:40.700986 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.701771 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.703263 (+ 1492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.703586 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.703649 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:40.704626 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:40.704627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:40.704627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:40.704632 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:40.706444 (+ 1812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:40.706783 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:40.706784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:40.706784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:40.706831 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:40.707000 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:40.707001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:40.707001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:40.707133 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:40.707191 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:40.707779 (+ 588us) server_negotiation.cc:299] Negotiation successful
0618 23:24:40.707908 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":21}
Jun 18 23:24:40 dist-test-slave-47qt krb5kdc[14411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825079, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.190@KRBTEST.COM for oryx/127.8.39.189@KRBTEST.COM
I20260618 23:24:40.716864 14993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:40.709550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42302 (local address 127.8.39.189:44675)
0618 23:24:40.709586 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:40.709588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:40.709725 (+ 137us) server_negotiation.cc:407] Connection header received
0618 23:24:40.709817 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:40.709818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:40.709946 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:40.710010 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:40.710472 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.711049 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.711883 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.712090 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.712177 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:40.714200 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:40.714202 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:40.714202 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:40.714210 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:40.715581 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:40.716013 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:40.716014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:40.716015 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:40.716075 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:40.716296 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:40.716297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:40.716297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:40.716419 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:40.716495 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:40.716683 (+ 188us) server_negotiation.cc:299] Negotiation successful
0618 23:24:40.716762 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260618 23:24:40.716878 15014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:40.709450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:44675 (local address 127.0.0.1:42302)
0618 23:24:40.709611 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:40.709655 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:40.709751 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:40.710017 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:40.710023 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:40.710325 (+ 302us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:40.710373 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:40.710375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.711171 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.711172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:40.711776 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:40.711778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.712181 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.712182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:40.712264 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:40.712756 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:40.712760 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:40.714112 (+ 1352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:40.715698 (+ 1586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:40.715705 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:40.715707 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:40.715911 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:40.716166 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:40.716167 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:40.716167 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:40.716232 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:40.716498 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:40.716505 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:40.716556 (+ 51us) client_negotiation.cc:769] Sending connection context
0618 23:24:40.716658 (+ 102us) client_negotiation.cc:240] Negotiation successful
0618 23:24:40.716772 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
I20260618 23:24:40.717461 14456 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.39.189:44675: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f"
member_type: NON_VOTER
last_known_addr {
host: "127.8.39.189"
port: 44675
}
attrs {
promote: true
}
}
dest_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b"
cas_config_opid_index: -1
I20260618 23:24:40.717775 14456 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: NON_VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: true } }
I20260618 23:24:40.718256 15016 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6b7ee65180f14c8cbe67b042b1685a8f (127.8.39.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: NON_VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: true } } }
I20260618 23:24:40.718657 14962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:40.720052 15016 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.39.189:44675
I20260618 23:24:40.720286 14962 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [term 1 FOLLOWER]: Refusing update from remote peer a3074d2dc0694e5fa8b0bfa6b3ff985b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260618 23:24:40.720253 15015 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: NON_VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: true } } }
I20260618 23:24:40.720329 15015 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:40.720319 14920 master.cc:537] Master@127.8.39.189:44675 shutting down...
I20260618 23:24:40.720620 15015 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: NON_VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260618 23:24:40.721412 14441 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260618 23:24:40.721951 14441 consensus_peers.cc:487] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b -> Peer 6b7ee65180f14c8cbe67b042b1685a8f (127.8.39.189:44675): Couldn't send request to peer 6b7ee65180f14c8cbe67b042b1685a8f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260618 23:24:40.723011 14920 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:24:40.723109 14920 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:40.723161 14920 tablet_replica.cc:330] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f: stopping tablet replica
I20260618 23:24:40.736348 14920 master.cc:552] Master@127.8.39.189:44675 shutdown complete.
I20260618 23:24:40.736413 14920 master_runner.cc:305] Clearing existing system tablet
I20260618 23:24:40.736544 14920 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:40.736603 14920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:40.737372 14920 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260618 23:24:40.737884 15020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:40.738026 14920 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:40.738087 14920 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:40.738113 14920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-1/wal
uuid: "6b7ee65180f14c8cbe67b042b1685a8f"
format_stamp: "Formatted at 2026-06-18 23:24:40 on dist-test-slave-47qt"
server_key: "3ffd758d87cf578047cb02fcf4cbd78c"
server_key_iv: "f80f6639c4bd8e4eb1f0d7050b9baf12"
server_key_version: "encryptionkey@0"
I20260618 23:24:40.738862 14920 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:40.740343 14920 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260618 23:24:40.740411 14920 log.cc:1196] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260618 23:24:40.740605 14920 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f: Deleting consensus metadata
I20260618 23:24:40.740727 14920 master_runner.cc:315] Copying system tablet from 127.8.39.190:41963
I20260618 23:24:40.741536 14920 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f: tablet copy: Beginning tablet copy session from remote peer at address 127.8.39.190:41963
I20260618 23:24:40.747767 14496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:40.741910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46848 (local address 127.8.39.190:41963)
0618 23:24:40.741957 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:40.741959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:40.742014 (+ 55us) server_negotiation.cc:407] Connection header received
0618 23:24:40.742084 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:40.742084 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:40.742092 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:40.742136 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:40.742487 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.742988 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.744077 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.744258 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.744320 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:40.745344 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:40.745345 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:40.745345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:40.745351 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:40.746763 (+ 1412us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:40.747125 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:40.747126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:40.747126 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:40.747173 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:40.747363 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:40.747363 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:40.747363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:40.747451 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:40.747498 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:40.747653 (+ 155us) server_negotiation.cc:299] Negotiation successful
0618 23:24:40.747707 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260618 23:24:40.747843 15026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:40.741814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.0.0.1:46848)
0618 23:24:40.741947 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:40.741958 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:40.742028 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:40.742151 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:40.742151 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:40.742348 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:40.742416 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:40.742417 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.743071 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.743072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:40.743952 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:40.743955 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.744336 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.744337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:40.744409 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:40.744865 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:40.744870 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:40.745269 (+ 399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:40.746842 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:40.746843 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:40.746843 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:40.747054 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:40.747235 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:40.747236 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:40.747237 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:40.747299 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:40.747509 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:40.747510 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:40.747554 (+ 44us) client_negotiation.cc:769] Sending connection context
0618 23:24:40.747637 (+ 83us) client_negotiation.cc:240] Negotiation successful
0618 23:24:40.747734 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":85,"threads_started":1}
I20260618 23:24:40.748414 14476 tablet_copy_service.cc:140] P a3074d2dc0694e5fa8b0bfa6b3ff985b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6b7ee65180f14c8cbe67b042b1685a8f ({username='oryx', principal='oryx/127.8.39.189@KRBTEST.COM'} at 127.0.0.1:46848)
I20260618 23:24:40.748514 14476 tablet_copy_service.cc:161] P a3074d2dc0694e5fa8b0bfa6b3ff985b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6b7ee65180f14c8cbe67b042b1685a8f at {username='oryx', principal='oryx/127.8.39.189@KRBTEST.COM'} at 127.0.0.1:46848: session id = 6b7ee65180f14c8cbe67b042b1685a8f-00000000000000000000000000000000
I20260618 23:24:40.749183 14476 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b: Tablet Copy: opened 0 blocks and 1 log segments
I20260618 23:24:40.749811 14920 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260618 23:24:40.751601 14920 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f: tablet copy: Starting download of 0 data blocks...
I20260618 23:24:40.751843 14920 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f: tablet copy: Starting download of 1 WAL segments...
I20260618 23:24:40.753270 14920 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260618 23:24:40.754438 14476 tablet_copy_service.cc:342] P a3074d2dc0694e5fa8b0bfa6b3ff985b: Request end of tablet copy session 6b7ee65180f14c8cbe67b042b1685a8f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.39.189@KRBTEST.COM'} at 127.0.0.1:46848
I20260618 23:24:40.754509 14476 tablet_copy_service.cc:433] P a3074d2dc0694e5fa8b0bfa6b3ff985b: ending tablet copy session 6b7ee65180f14c8cbe67b042b1685a8f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6b7ee65180f14c8cbe67b042b1685a8f
I20260618 23:24:40.757234 14920 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:40.758631 15027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:40.758625 15028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:40.758625 15030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:40.758937 14920 server_base.cc:1034] running on GCE node
I20260618 23:24:40.759140 14920 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:40.759445 14920 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:40.760617 14920 hybrid_clock.cc:648] HybridClock initialized: now 1781825080760608 us; error 26 us; skew 500 ppm
Jun 18 23:24:40 dist-test-slave-47qt krb5kdc[14411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825080, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:40.762113 14920 init.cc:377] Logged in from keytab as oryx/127.8.39.189@KRBTEST.COM (short username oryx)
I20260618 23:24:40.762404 14920 webserver.cc:460] Webserver started at http://127.8.39.189:42253/ using document root <none> and password file <none>
I20260618 23:24:40.762532 14920 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:40.762591 14920 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:40.763200 14920 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260618 23:24:40.763747 15037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:40.763906 14920 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260618 23:24:40.763964 14920 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-1/wal
uuid: "6b7ee65180f14c8cbe67b042b1685a8f"
format_stamp: "Formatted at 2026-06-18 23:24:40 on dist-test-slave-47qt"
server_key: "3ffd758d87cf578047cb02fcf4cbd78c"
server_key_iv: "f80f6639c4bd8e4eb1f0d7050b9baf12"
server_key_version: "encryptionkey@0"
I20260618 23:24:40.764022 14920 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:40.784502 14920 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:40.786218 14920 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:40.789436 14920 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.189:44675
I20260618 23:24:40.789577 15089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.189:44675 every 8 connection(s)
I20260618 23:24:40.789845 14920 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-1/data/info.pb
I20260618 23:24:40.790910 15090 sys_catalog.cc:263] Verifying existing consensus state
I20260618 23:24:40.791347 15090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f: Bootstrap starting.
I20260618 23:24:40.793767 15090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 23:24:40.794013 15090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f: Bootstrap complete.
I20260618 23:24:40.794224 15090 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: NON_VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: true } }
I20260618 23:24:40.794328 15090 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6b7ee65180f14c8cbe67b042b1685a8f, State: Initialized, Role: LEARNER
I20260618 23:24:40.794376 15090 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: NON_VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: true } }
I20260618 23:24:40.794562 15090 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:24:40.794555 15093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: NON_VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: true } } }
I20260618 23:24:40.794672 15093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [sys.catalog]: This master's current role is: LEARNER
I20260618 23:24:40.796168 15104 catalog_manager.cc:1259] Loaded cluster ID: 05a830b28cdd4f759c72bf6b6c1a803a
I20260618 23:24:40.796222 15104 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f: loading cluster ID for follower catalog manager: success
I20260618 23:24:40.797251 15104 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f: acquiring CA information for follower catalog manager: success
I20260618 23:24:40.797464 15104 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260618 23:24:40.843170 15105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:40.835308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42334 (local address 127.8.39.189:44675)
0618 23:24:40.835453 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:40.835454 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:40.835462 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:24:40.835485 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:40.835485 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:40.835495 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:40.835562 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:40.836252 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.836916 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.837968 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:40.838202 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:40.838259 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:40.839587 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:40.839588 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:40.839589 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:40.839594 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:40.841519 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:40.841984 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:40.841985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:40.841985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:40.842094 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:40.842384 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:40.842385 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:40.842385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:40.842551 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:40.842661 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:40.842863 (+ 202us) server_negotiation.cc:299] Negotiation successful
0618 23:24:40.842996 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":95,"threads_started":1}
WARNING: no policy specified for oryx/127.8.39.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.39.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.39.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.188.keytab.
Entry for principal oryx/127.8.39.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.188.keytab.
I20260618 23:24:40.868883 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.188:37721
--webserver_interface=127.8.39.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.188.keytab
--principal=oryx/127.8.39.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:41963,127.8.39.189:44675,127.8.39.188:37721
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:40.953419 15108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:40.953624 15108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:40.953646 15108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:40.954885 15108 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:24:40.954932 15108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:40.954947 15108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:40.954960 15108 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:24:40.954972 15108 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:24:40.956575 15108 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:41963,127.8.39.189:44675,127.8.39.188:37721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.188.keytab
--principal=oryx/127.8.39.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.188:37721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.39.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:40.956841 15108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:40.957057 15108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:40.962270 15114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:40.962261 15113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:40.962302 15116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:40.962522 15108 server_base.cc:1034] running on GCE node
I20260618 23:24:40.962786 15108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:40.963141 15108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:40.964334 15108 hybrid_clock.cc:648] HybridClock initialized: now 1781825080964280 us; error 72 us; skew 500 ppm
Jun 18 23:24:40 dist-test-slave-47qt krb5kdc[14411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825080, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:40.967038 15108 init.cc:377] Logged in from keytab as oryx/127.8.39.188@KRBTEST.COM (short username oryx)
I20260618 23:24:40.967468 15108 webserver.cc:460] Webserver started at http://127.8.39.188:35927/ using document root <none> and password file <none>
I20260618 23:24:40.967694 15108 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:40.967764 15108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:40.967912 15108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:40.968983 15108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-2/data/instance:
uuid: "425e3e692fa8473b9a729d973f033081"
format_stamp: "Formatted at 2026-06-18 23:24:40 on dist-test-slave-47qt"
server_key: "948d9d969d8dda41e6ba25d775ff4bbf"
server_key_iv: "cabd1c48ef488a9239bea228bd14bb86"
server_key_version: "encryptionkey@0"
I20260618 23:24:40.969362 15108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-2/wal/instance:
uuid: "425e3e692fa8473b9a729d973f033081"
format_stamp: "Formatted at 2026-06-18 23:24:40 on dist-test-slave-47qt"
server_key: "948d9d969d8dda41e6ba25d775ff4bbf"
server_key_iv: "cabd1c48ef488a9239bea228bd14bb86"
server_key_version: "encryptionkey@0"
I20260618 23:24:40.970685 15108 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:24:40.971414 15123 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:40.971572 15108 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260618 23:24:40.971668 15108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-2/wal
uuid: "425e3e692fa8473b9a729d973f033081"
format_stamp: "Formatted at 2026-06-18 23:24:40 on dist-test-slave-47qt"
server_key: "948d9d969d8dda41e6ba25d775ff4bbf"
server_key_iv: "cabd1c48ef488a9239bea228bd14bb86"
server_key_version: "encryptionkey@0"
I20260618 23:24:40.971750 15108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:41.002995 15108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:41.005404 15108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:41.005599 15108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:41.009946 15108 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.188:37721
I20260618 23:24:41.010013 15175 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.188:37721 every 8 connection(s)
I20260618 23:24:41.010377 15108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-2/data/info.pb
I20260618 23:24:41.011243 15176 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:41.012732 15176 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:24:41.013881 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 15108
Jun 18 23:24:41 dist-test-slave-47qt krb5kdc[14411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825080, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.188@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM
Jun 18 23:24:41 dist-test-slave-47qt krb5kdc[14411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.39.188@KRBTEST.COM
I20260618 23:24:41.023859 14496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.013546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46852 (local address 127.8.39.190:41963)
0618 23:24:41.013626 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:41.013628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:41.013790 (+ 162us) server_negotiation.cc:407] Connection header received
0618 23:24:41.014441 (+ 651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:41.014443 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:41.014456 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:41.014533 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:41.015274 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.015987 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.017601 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.017818 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.017900 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.020187 (+ 2287us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:41.020188 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:41.020188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:41.020195 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:41.022317 (+ 2122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.022776 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.022778 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.022778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.022839 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.023179 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.023180 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.023180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.023310 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:41.023394 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:41.023672 (+ 278us) server_negotiation.cc:299] Negotiation successful
0618 23:24:41.023769 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 23:24:41.024025 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.013425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.0.0.1:46852)
0618 23:24:41.013658 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:41.013709 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:41.014317 (+ 608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:41.014611 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:41.014616 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:41.014934 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:41.015174 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.015179 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.016207 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.016209 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.017491 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.017494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.017922 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.017923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.018026 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.018596 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:41.018606 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:41.020095 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:41.022417 (+ 2322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.022422 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.022424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.022693 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.022911 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.022911 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.022912 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.023008 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.023384 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:41.023390 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:41.023522 (+ 132us) client_negotiation.cc:769] Sending connection context
0618 23:24:41.023636 (+ 114us) client_negotiation.cc:240] Negotiation successful
0618 23:24:41.023791 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":138,"threads_started":1}
I20260618 23:24:41.024507 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.014570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37644 (local address 127.8.39.188:37721)
0618 23:24:41.014744 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:41.014747 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:41.014764 (+ 17us) server_negotiation.cc:407] Connection header received
0618 23:24:41.014783 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:41.014784 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:41.014807 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:41.014883 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:41.015560 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.016655 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.017848 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.018202 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.018288 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.020775 (+ 2487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:41.020782 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:41.020782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:41.020789 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:41.022889 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.023227 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.023229 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.023230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.023288 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.023488 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.023489 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.023490 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.023632 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:41.023799 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:41.023842 (+ 43us) server_negotiation.cc:299] Negotiation successful
0618 23:24:41.024383 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":91,"threads_started":1}
I20260618 23:24:41.024582 15176 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 23:24:41 dist-test-slave-47qt krb5kdc[14411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825080, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.188@KRBTEST.COM for oryx/127.8.39.189@KRBTEST.COM
I20260618 23:24:41.030637 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.024862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:44675 (local address 127.0.0.1:42348)
0618 23:24:41.024904 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:41.024913 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:41.024944 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:41.025163 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:41.025164 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:41.025293 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:41.025321 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.025323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.025977 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.025978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.026478 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.026487 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.026873 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.026874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.026934 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.027259 (+ 325us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:41.027263 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:41.028361 (+ 1098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:41.029841 (+ 1480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.029842 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.029842 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.030042 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.030213 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.030213 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.030213 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.030260 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.030431 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:41.030431 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:41.030464 (+ 33us) client_negotiation.cc:769] Sending connection context
0618 23:24:41.030515 (+ 51us) client_negotiation.cc:240] Negotiation successful
0618 23:24:41.030577 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260618 23:24:41.030656 15105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.024939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42348 (local address 127.8.39.189:44675)
0618 23:24:41.025010 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:41.025012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:41.025019 (+ 7us) server_negotiation.cc:407] Connection header received
0618 23:24:41.025049 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:41.025051 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:41.025062 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:41.025129 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:41.025408 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.025886 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.026581 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.026788 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.026872 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.028440 (+ 1568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:41.028442 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:41.028442 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:41.028449 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:41.029770 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.030109 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.030110 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.030110 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.030155 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.030297 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.030297 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.030298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.030382 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:41.030425 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:41.030529 (+ 104us) server_negotiation.cc:299] Negotiation successful
0618 23:24:41.030587 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 23:24:41.031078 15176 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.39.188" port: 37721 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 23:24:41 dist-test-slave-47qt krb5kdc[14411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825080, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.188@KRBTEST.COM for oryx/127.8.39.188@KRBTEST.COM
I20260618 23:24:41.037883 15178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.031347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:37721 (local address 127.0.0.1:37658)
0618 23:24:41.031397 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:41.031407 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:41.031479 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:41.031608 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:41.031609 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:41.031729 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:41.031751 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.031752 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.032577 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.032577 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.033243 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.033245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.033619 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.033620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.033683 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.034043 (+ 360us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:41.034047 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:41.035143 (+ 1096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:41.036664 (+ 1521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.036665 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.036665 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.036846 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.037288 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.037289 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.037290 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.037363 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.037663 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:41.037663 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:41.037696 (+ 33us) client_negotiation.cc:769] Sending connection context
0618 23:24:41.037748 (+ 52us) client_negotiation.cc:240] Negotiation successful
0618 23:24:41.037823 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260618 23:24:41.037945 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.031418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37658 (local address 127.8.39.188:37721)
0618 23:24:41.031483 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:41.031485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:41.031493 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:24:41.031541 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:41.031542 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:41.031552 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:41.031605 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:41.031823 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.032375 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.033316 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.033536 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.033620 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.035235 (+ 1615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:41.035236 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:41.035236 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:41.035243 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:41.036533 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.036937 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.036938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.036938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.036990 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.037466 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.037467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.037467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.037592 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:41.037655 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:41.037785 (+ 130us) server_negotiation.cc:299] Negotiation successful
0618 23:24:41.037870 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":50}
I20260618 23:24:41.039353 15176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081: Bootstrap starting.
I20260618 23:24:41.040153 15176 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:41.040422 15176 log.cc:824] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:41.041108 15176 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081: No bootstrap required, opened a new log
I20260618 23:24:41.042507 15176 raft_consensus.cc:348] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 37721 } }
I20260618 23:24:41.042649 15176 raft_consensus.cc:374] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:41.042698 15176 raft_consensus.cc:729] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 425e3e692fa8473b9a729d973f033081, State: Initialized, Role: FOLLOWER
I20260618 23:24:41.042824 15176 consensus_queue.cc:260] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 37721 } }
I20260618 23:24:41.043061 15181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 37721 } } }
I20260618 23:24:41.043138 15176 sys_catalog.cc:564] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:24:41.043159 15181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [sys.catalog]: This master's current role is: FOLLOWER
W20260618 23:24:41.044667 15192 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 23:24:41.044746 15192 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:24:41.047343 15108 master_runner.cc:428] Detected that this master 425e3e692fa8473b9a729d973f033081 is joining an existing cluster
I20260618 23:24:41.047411 15108 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.39.188:37721
I20260618 23:24:41.055529 15199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.048657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:37721 (local address 127.0.0.1:37666)
0618 23:24:41.048782 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:41.048791 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:41.048858 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:41.049005 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:41.049005 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:41.049291 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:41.049349 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.049351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.050223 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.050223 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.051081 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.051083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.051562 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.051563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.051617 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.052008 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:41.052012 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:41.052441 (+ 429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:41.054323 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.054325 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.054325 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.054568 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.054839 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.054839 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.054840 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.054907 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.055173 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:41.055174 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:41.055229 (+ 55us) client_negotiation.cc:769] Sending connection context
0618 23:24:41.055300 (+ 71us) client_negotiation.cc:240] Negotiation successful
0618 23:24:41.055397 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20260618 23:24:41.055769 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.048616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37666 (local address 127.8.39.188:37721)
0618 23:24:41.048678 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:41.048680 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:41.048838 (+ 158us) server_negotiation.cc:407] Connection header received
0618 23:24:41.048920 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:41.048921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:41.048928 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:41.048955 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:41.049429 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.050125 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.051171 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.051442 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.051489 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.052566 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:41.052567 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:41.052568 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:41.052573 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:41.054220 (+ 1647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.054707 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.054708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.054708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.054761 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.054985 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.054985 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.054985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.055090 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:41.055157 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:41.055522 (+ 365us) server_negotiation.cc:299] Negotiation successful
0618 23:24:41.055646 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 23:24:41.074214 15200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.048365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:44675 (local address 127.0.0.1:42364)
0618 23:24:41.048758 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:41.048767 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:41.048831 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:41.048995 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:41.048995 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:41.049251 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:41.049320 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.049322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.050192 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.050193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.051070 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.051071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.051597 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.051598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.051652 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.070362 (+ 18710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:41.070367 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:41.070759 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:41.072513 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.072514 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.072515 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.072818 (+ 303us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.073209 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.073210 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.073210 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.073294 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.073745 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:41.073746 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:41.073790 (+ 44us) client_negotiation.cc:769] Sending connection context
0618 23:24:41.073890 (+ 100us) client_negotiation.cc:240] Negotiation successful
0618 23:24:41.074012 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":4,"thread_start_us":78,"threads_started":1}
I20260618 23:24:41.074249 15105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.048606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42364 (local address 127.8.39.189:44675)
0618 23:24:41.048647 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:41.048649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:41.048809 (+ 160us) server_negotiation.cc:407] Connection header received
0618 23:24:41.048897 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:41.048898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:41.048907 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:41.048935 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:41.049404 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.050110 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.051167 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.051464 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.051502 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.070854 (+ 19352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:41.070856 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:41.070856 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:41.070864 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:41.072399 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.072903 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.072904 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.072904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.072964 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.073428 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.073429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.073430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.073624 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:41.073737 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:41.073947 (+ 210us) server_negotiation.cc:299] Negotiation successful
0618 23:24:41.074154 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260618 23:24:41.074575 14496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.048370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46862 (local address 127.8.39.190:41963)
0618 23:24:41.048545 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:41.048547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:41.048565 (+ 18us) server_negotiation.cc:407] Connection header received
0618 23:24:41.048642 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:41.048643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:41.048655 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:41.048687 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:41.049184 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.049960 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.051167 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.051464 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.051502 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.070701 (+ 19199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:41.070704 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:41.070704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:41.070714 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:41.072540 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.072945 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.072946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.072947 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.073001 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.073774 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.073775 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.073775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.073952 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:41.074023 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:41.074390 (+ 367us) server_negotiation.cc:299] Negotiation successful
0618 23:24:41.074482 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152}
I20260618 23:24:41.074909 15198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.048360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.0.0.1:46862)
0618 23:24:41.048498 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:41.048508 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:41.048562 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:41.048751 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:41.048752 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:41.048961 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:41.049039 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.049040 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.050047 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.050058 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.051070 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.051071 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.051590 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.051591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.051652 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.069985 (+ 18333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:41.069992 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:41.070560 (+ 568us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:41.072630 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.072630 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.072631 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.072888 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.073220 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.073220 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.073221 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.073305 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.074056 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:41.074057 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:41.074192 (+ 135us) client_negotiation.cc:769] Sending connection context
0618 23:24:41.074697 (+ 505us) client_negotiation.cc:240] Negotiation successful
0618 23:24:41.074802 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":33,"thread_start_us":94,"threads_started":1}
I20260618 23:24:41.075881 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 14518
I20260618 23:24:41.082677 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.129:45209
--local_ip_for_outbound_sockets=127.8.39.129
--tserver_master_addrs=127.8.39.190:41963
--webserver_port=35711
--webserver_interface=127.8.39.129
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.129.keytab
--principal=oryx/127.8.39.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.39.190:41963,127.8.39.189:44675
--tserver_master_addrs=127.8.39.190:41963,127.8.39.189:44675,127.8.39.188:37721 with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 23:24:41 dist-test-slave-47qt krb5kdc[14411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825079, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.190@KRBTEST.COM for oryx/127.8.39.188@KRBTEST.COM
I20260618 23:24:41.086556 15014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.078550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:37721 (local address 127.0.0.1:37674)
0618 23:24:41.078608 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:41.078619 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:41.078653 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:41.078812 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:41.078813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:41.078997 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:41.079049 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.079051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.079968 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.079969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.080885 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.080887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.081460 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.081461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.081535 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.081957 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:41.081963 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:41.083697 (+ 1734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:41.085578 (+ 1881us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.085580 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.085580 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.085775 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.085951 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.085951 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.085952 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.086001 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.086263 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:41.086264 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:41.086306 (+ 42us) client_negotiation.cc:769] Sending connection context
0618 23:24:41.086362 (+ 56us) client_negotiation.cc:240] Negotiation successful
0618 23:24:41.086459 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260618 23:24:41.086807 15179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.078604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37674 (local address 127.8.39.188:37721)
0618 23:24:41.078651 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:41.078653 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:41.078663 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:24:41.078719 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:41.078720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:41.078730 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:41.078791 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:41.079127 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.079858 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.080988 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.081338 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.081406 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.083795 (+ 2389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:41.083796 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:41.083797 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:41.083803 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:41.085488 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.085844 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.085845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.085846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.085893 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.086069 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.086070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.086070 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.086189 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:41.086240 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:41.086661 (+ 421us) server_negotiation.cc:299] Negotiation successful
0618 23:24:41.086728 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260618 23:24:41.088188 14456 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.39.188:37721: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "425e3e692fa8473b9a729d973f033081"
member_type: NON_VOTER
last_known_addr {
host: "127.8.39.188"
port: 37721
}
attrs {
promote: true
}
}
dest_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b"
cas_config_opid_index: 5
I20260618 23:24:41.088454 14456 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: NON_VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: true } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: NON_VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: true } }
I20260618 23:24:41.088792 15015 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 425e3e692fa8473b9a729d973f033081 (127.8.39.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: NON_VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: true } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: NON_VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: true } } }
I20260618 23:24:41.089066 15150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:41.089829 15015 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.39.188:37721
I20260618 23:24:41.089896 15015 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: NON_VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: true } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: NON_VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: true } } }
I20260618 23:24:41.089957 15015 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:41.089987 15150 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [term 1 FOLLOWER]: Refusing update from remote peer a3074d2dc0694e5fa8b0bfa6b3ff985b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260618 23:24:41.090328 15016 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [LEADER]: Connected to new peer: Peer: permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: NON_VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:41.090704 15108 master.cc:537] Master@127.8.39.188:37721 shutting down...
I20260618 23:24:41.093490 15150 raft_consensus.cc:858] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 37721 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: NON_VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: NON_VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: true } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: NON_VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: true } }
W20260618 23:24:41.093560 15150 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 37721 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: NON_VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: NON_VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: true } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: NON_VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: true } }
I20260618 23:24:41.094019 15150 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6b7ee65180f14c8cbe67b042b1685a8f (127.8.39.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: NON_VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: true } } }
I20260618 23:24:41.094980 15014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.090161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:44675 (local address 127.0.0.1:42376)
0618 23:24:41.090492 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:41.090504 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:41.090549 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:41.090771 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:41.090772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:41.090774 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:41.090817 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.090819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.091816 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.091817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.093867 (+ 2050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.093870 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.094706 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.094708 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.094781 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.094801 (+ 20us) client_negotiation.cc:769] Sending connection context
0618 23:24:41.094856 (+ 55us) client_negotiation.cc:240] Negotiation successful
0618 23:24:41.094901 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317}
I20260618 23:24:41.094978 15181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3074d2dc0694e5fa8b0bfa6b3ff985b. Latest consensus state: current_term: 1 leader_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: NON_VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: true } } }
I20260618 23:24:41.095193 15181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260618 23:24:41.095288 15181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: NON_VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: true } } }
I20260618 23:24:41.095353 15181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260618 23:24:41.095383 15181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: NON_VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: true } } }
I20260618 23:24:41.095422 15181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260618 23:24:41.095432 15105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.090261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42376 (local address 127.8.39.189:44675)
0618 23:24:41.090463 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:41.090465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:41.090602 (+ 137us) server_negotiation.cc:407] Connection header received
0618 23:24:41.090629 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:41.090630 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:41.090655 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:41.090711 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:41.090887 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.091725 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.093965 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.094534 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.094606 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.094673 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:41.095282 (+ 609us) server_negotiation.cc:299] Negotiation successful
0618 23:24:41.095358 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176}
I20260618 23:24:41.095638 15064 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [term 1 LEARNER]: Refusing update from remote peer a3074d2dc0694e5fa8b0bfa6b3ff985b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260618 23:24:41.095880 15016 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: NON_VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260618 23:24:41.096403 15064 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 425e3e692fa8473b9a729d973f033081 (127.8.39.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: NON_VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: true } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: NON_VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: true } } }
I20260618 23:24:41.097585 15202 mvcc.cc:205] Tried to move back new op lower bound from 7298355527724597248 to 7298355527663988736. Current Snapshot: MvccSnapshot[applied={T|T < 7298355527674314752}]
I20260618 23:24:41.097839 15093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3074d2dc0694e5fa8b0bfa6b3ff985b. Latest consensus state: current_term: 1 leader_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: NON_VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: true } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: NON_VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: true } } }
I20260618 23:24:41.097927 15093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [sys.catalog]: This master's current role is: LEARNER
I20260618 23:24:41.097970 15093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: NON_VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: true } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: NON_VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: true } } }
I20260618 23:24:41.098014 15093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [sys.catalog]: This master's current role is: LEARNER
I20260618 23:24:41.098232 15016 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: NON_VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: true } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: NON_VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: true } } }
I20260618 23:24:41.098322 15016 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:41.098326 15108 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260618 23:24:41.098412 15108 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260618 23:24:41.098449 15108 tablet_replica.cc:330] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081: stopping tablet replica
W20260618 23:24:41.098826 14442 consensus_peers.cc:487] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b -> Peer 425e3e692fa8473b9a729d973f033081 (127.8.39.188:37721): Couldn't send request to peer 425e3e692fa8473b9a729d973f033081. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260618 23:24:41.101608 15016 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b: attempting to promote NON_VOTER 6b7ee65180f14c8cbe67b042b1685a8f to VOTER
I20260618 23:24:41.101804 15016 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: NON_VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: true } }
I20260618 23:24:41.102346 15064 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [term 1 LEARNER]: Refusing update from remote peer a3074d2dc0694e5fa8b0bfa6b3ff985b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260618 23:24:41.102380 15014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.102223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:37721 (local address 127.0.0.1:37690)
0618 23:24:41.102311 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:41.102351 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260618 23:24:41.102653 15211 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260618 23:24:41.102787 15014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.102702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:37721 (local address 127.0.0.1:37706)
0618 23:24:41.102738 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:41.102761 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260618 23:24:41.103022 14442 consensus_peers.cc:487] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b -> Peer 425e3e692fa8473b9a729d973f033081 (127.8.39.188:37721): Couldn't send request to peer 425e3e692fa8473b9a729d973f033081. Status: Network error: Client connection negotiation failed: client connection to 127.8.39.188:37721: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260618 23:24:41.103245 15015 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6b7ee65180f14c8cbe67b042b1685a8f (127.8.39.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: NON_VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: true } } }
I20260618 23:24:41.103437 15064 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6b7ee65180f14c8cbe67b042b1685a8f (127.8.39.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: NON_VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: true } } }
I20260618 23:24:41.104177 15015 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: NON_VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: true } } }
I20260618 23:24:41.104260 15015 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:41.104317 15015 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: NON_VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: true } } }
I20260618 23:24:41.104377 15015 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:41.104398 15209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: NON_VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: true } } }
I20260618 23:24:41.104530 15209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:24:41.116529 15108 master.cc:552] Master@127.8.39.188:37721 shutdown complete.
I20260618 23:24:41.116604 15108 master_runner.cc:305] Clearing existing system tablet
I20260618 23:24:41.116734 15108 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:41.116796 15108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:41.117569 15108 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260618 23:24:41.118059 15214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:41.118180 15108 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:41.118224 15108 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:41.118252 15108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-2/wal
uuid: "425e3e692fa8473b9a729d973f033081"
format_stamp: "Formatted at 2026-06-18 23:24:40 on dist-test-slave-47qt"
server_key: "948d9d969d8dda41e6ba25d775ff4bbf"
server_key_iv: "cabd1c48ef488a9239bea228bd14bb86"
server_key_version: "encryptionkey@0"
I20260618 23:24:41.118908 15108 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:41.120278 15108 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260618 23:24:41.120364 15108 log.cc:1196] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260618 23:24:41.120594 15108 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081: Deleting consensus metadata
I20260618 23:24:41.120741 15108 master_runner.cc:315] Copying system tablet from 127.8.39.190:41963
I20260618 23:24:41.121408 15108 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081: tablet copy: Beginning tablet copy session from remote peer at address 127.8.39.190:41963
I20260618 23:24:41.128103 14496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.121749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46872 (local address 127.8.39.190:41963)
0618 23:24:41.121794 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:41.121796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:41.121905 (+ 109us) server_negotiation.cc:407] Connection header received
0618 23:24:41.121984 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:41.121985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:41.121995 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:41.122046 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:41.122342 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.122848 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.124012 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.124208 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.124293 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.125337 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:41.125338 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:41.125338 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:41.125345 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:41.126949 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.127318 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.127319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.127319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.127376 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.127587 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.127588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.127588 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.127703 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:41.127782 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:41.127935 (+ 153us) server_negotiation.cc:299] Negotiation successful
0618 23:24:41.128011 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260618 23:24:41.128115 15220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.121712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.0.0.1:46872)
0618 23:24:41.121827 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:41.121836 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:41.121920 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:41.122048 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:41.122049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:41.122220 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:41.122263 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.122265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.122934 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.122935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.123907 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.123909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.124295 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.124296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.124356 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.124865 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:41.124870 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:41.125236 (+ 366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:41.127063 (+ 1827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.127065 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.127065 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.127241 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.127449 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.127450 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.127451 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.127507 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.127774 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:41.127775 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:41.127819 (+ 44us) client_negotiation.cc:769] Sending connection context
0618 23:24:41.127911 (+ 92us) client_negotiation.cc:240] Negotiation successful
0618 23:24:41.128011 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20260618 23:24:41.128724 14476 tablet_copy_service.cc:140] P a3074d2dc0694e5fa8b0bfa6b3ff985b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 425e3e692fa8473b9a729d973f033081 ({username='oryx', principal='oryx/127.8.39.188@KRBTEST.COM'} at 127.0.0.1:46872)
I20260618 23:24:41.128808 14476 tablet_copy_service.cc:161] P a3074d2dc0694e5fa8b0bfa6b3ff985b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 425e3e692fa8473b9a729d973f033081 at {username='oryx', principal='oryx/127.8.39.188@KRBTEST.COM'} at 127.0.0.1:46872: session id = 425e3e692fa8473b9a729d973f033081-00000000000000000000000000000000
I20260618 23:24:41.129362 14476 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b: Tablet Copy: opened 0 blocks and 1 log segments
I20260618 23:24:41.129806 15108 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260618 23:24:41.130746 15108 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081: tablet copy: Starting download of 0 data blocks...
I20260618 23:24:41.130903 15108 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081: tablet copy: Starting download of 1 WAL segments...
I20260618 23:24:41.131551 15108 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260618 23:24:41.132268 14476 tablet_copy_service.cc:342] P a3074d2dc0694e5fa8b0bfa6b3ff985b: Request end of tablet copy session 425e3e692fa8473b9a729d973f033081-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.39.188@KRBTEST.COM'} at 127.0.0.1:46872
I20260618 23:24:41.132318 14476 tablet_copy_service.cc:433] P a3074d2dc0694e5fa8b0bfa6b3ff985b: ending tablet copy session 425e3e692fa8473b9a729d973f033081-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 425e3e692fa8473b9a729d973f033081
I20260618 23:24:41.134923 15108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:41.136528 15221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:41.137478 15108 server_base.cc:1034] running on GCE node
W20260618 23:24:41.138230 15224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:41.142081 15222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:41.142289 15108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:41.142457 15108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:41.143596 15108 hybrid_clock.cc:648] HybridClock initialized: now 1781825081143585 us; error 34 us; skew 500 ppm
Jun 18 23:24:41 dist-test-slave-47qt krb5kdc[14411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825081, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:41.144939 15108 init.cc:377] Logged in from keytab as oryx/127.8.39.188@KRBTEST.COM (short username oryx)
I20260618 23:24:41.145251 15108 webserver.cc:460] Webserver started at http://127.8.39.188:33971/ using document root <none> and password file <none>
I20260618 23:24:41.145370 15108 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:41.145406 15108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:41.146030 15108 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260618 23:24:41.146543 15231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:41.146673 15108 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260618 23:24:41.146719 15108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-2/wal
uuid: "425e3e692fa8473b9a729d973f033081"
format_stamp: "Formatted at 2026-06-18 23:24:40 on dist-test-slave-47qt"
server_key: "948d9d969d8dda41e6ba25d775ff4bbf"
server_key_iv: "cabd1c48ef488a9239bea228bd14bb86"
server_key_version: "encryptionkey@0"
I20260618 23:24:41.146780 15108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:41.155872 15108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:41.157649 15108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:41.160919 15108 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.188:37721
I20260618 23:24:41.161303 15108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/master-2/data/info.pb
I20260618 23:24:41.161641 15283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.188:37721 every 8 connection(s)
I20260618 23:24:41.162213 15284 sys_catalog.cc:263] Verifying existing consensus state
I20260618 23:24:41.162609 15284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081: Bootstrap starting.
I20260618 23:24:41.164772 15284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 23:24:41.165014 15284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081: Bootstrap complete.
I20260618 23:24:41.165267 15284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: NON_VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: true } }
I20260618 23:24:41.165359 15284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 425e3e692fa8473b9a729d973f033081, State: Initialized, Role: LEARNER
I20260618 23:24:41.165403 15284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: NON_VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: true } }
I20260618 23:24:41.165571 15284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:24:41.165571 15287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: NON_VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: true } } }
I20260618 23:24:41.165802 15287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [sys.catalog]: This master's current role is: LEARNER
I20260618 23:24:41.167004 15298 catalog_manager.cc:1259] Loaded cluster ID: 05a830b28cdd4f759c72bf6b6c1a803a
I20260618 23:24:41.167086 15298 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081: loading cluster ID for follower catalog manager: success
I20260618 23:24:41.168223 15298 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081: acquiring CA information for follower catalog manager: success
I20260618 23:24:41.168504 15298 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260618 23:24:41.204210 15201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:41.204528 15201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:41.204571 15201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:41.206387 15201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:41.206463 15201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:41.206507 15201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.129
I20260618 23:24:41.208806 15201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.129.keytab
--principal=oryx/127.8.39.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.129:45209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.39.129
--webserver_port=35711
--tserver_master_addrs=127.8.39.190:41963,127.8.39.189:44675,127.8.39.188:37721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.129
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:41.209074 15201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:41.209388 15201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:41.214933 15304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:41.215134 15303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:41.215158 15306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:41.215552 15201 server_base.cc:1034] running on GCE node
I20260618 23:24:41.215763 15201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:41.216032 15201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:41.217193 15201 hybrid_clock.cc:648] HybridClock initialized: now 1781825081217183 us; error 32 us; skew 500 ppm
Jun 18 23:24:41 dist-test-slave-47qt krb5kdc[14411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825081, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:41.220140 15201 init.cc:377] Logged in from keytab as oryx/127.8.39.129@KRBTEST.COM (short username oryx)
I20260618 23:24:41.220556 15201 webserver.cc:460] Webserver started at http://127.8.39.129:35711/ using document root <none> and password file <none>
I20260618 23:24:41.220813 15201 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:41.220903 15201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:41.222365 15201 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:41.223052 15313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:41.223233 15201 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:41.223331 15201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-0/wal
uuid: "b46d97a84c2b4d88b79434d5e2bf1a16"
format_stamp: "Formatted at 2026-06-18 23:24:40 on dist-test-slave-47qt"
server_key: "693362169c722a21453370b52e8ce402"
server_key_iv: "2217625ad8d9402d55b5c43155847c8e"
server_key_version: "encryptionkey@0"
I20260618 23:24:41.223686 15201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:41.246960 15201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:41.249280 15201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:41.249423 15201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:41.249680 15201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:41.249992 15201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:41.250051 15201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:41.250092 15201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:41.250118 15201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:41.255539 15201 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.129:45209
I20260618 23:24:41.255620 15426 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.129:45209 every 8 connection(s)
I20260618 23:24:41.255944 15201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
I20260618 23:24:41.261231 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 15201
I20260618 23:24:41.261382 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 14652
Jun 18 23:24:41 dist-test-slave-47qt krb5kdc[14411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825081, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.189@KRBTEST.COM
Jun 18 23:24:41 dist-test-slave-47qt krb5kdc[14411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825081, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.188@KRBTEST.COM
Jun 18 23:24:41 dist-test-slave-47qt krb5kdc[14411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825081, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM
I20260618 23:24:41.268311 15105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.256995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:50699 (local address 127.8.39.189:44675)
0618 23:24:41.257056 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:41.257058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:41.257192 (+ 134us) server_negotiation.cc:407] Connection header received
0618 23:24:41.257911 (+ 719us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:41.257911 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:41.257919 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:41.257985 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:41.258672 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.259177 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.260621 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.260919 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.260958 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.264572 (+ 3614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:41.264574 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:41.264574 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:41.264583 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:41.266552 (+ 1969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.267159 (+ 607us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.267160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.267160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.267228 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.267532 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.267533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.267533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.267683 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:41.267725 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:41.268106 (+ 381us) server_negotiation.cc:299] Negotiation successful
0618 23:24:41.268191 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260618 23:24:41.268977 15432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.256866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:44675 (local address 127.8.39.129:50699)
0618 23:24:41.257110 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:41.257137 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:41.257669 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:41.257991 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:41.257992 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:41.258366 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:41.258577 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.258582 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.259287 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.259289 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.260518 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.260520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.261165 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.261166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.261347 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.262512 (+ 1165us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:41.262517 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:41.264441 (+ 1924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:41.266648 (+ 2207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.266655 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.266658 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.267064 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.267307 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.267308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.267308 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.267436 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.267772 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:41.267779 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:41.267942 (+ 163us) client_negotiation.cc:769] Sending connection context
0618 23:24:41.268545 (+ 603us) client_negotiation.cc:240] Negotiation successful
0618 23:24:41.268726 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":24,"thread_start_us":82,"threads_started":1}
I20260618 23:24:41.269048 14496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.256688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:42565 (local address 127.8.39.190:41963)
0618 23:24:41.256731 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:41.256733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:41.257033 (+ 300us) server_negotiation.cc:407] Connection header received
0618 23:24:41.257790 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:41.257791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:41.257805 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:41.257864 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:41.258695 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.259458 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.260621 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.260916 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.260958 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.265977 (+ 5019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:41.265979 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:41.265980 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:41.265989 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:41.267671 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.268195 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.268196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.268197 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.268252 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.268436 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.268436 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.268437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.268566 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:41.268609 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:41.268807 (+ 198us) server_negotiation.cc:299] Negotiation successful
0618 23:24:41.268891 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260618 23:24:41.269421 15431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.256656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.8.39.129:42565)
0618 23:24:41.256789 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:41.256832 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:41.257660 (+ 828us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:41.257929 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:41.257932 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:41.258557 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:41.258607 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.258608 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.259542 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.259543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.260518 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.260520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.261178 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.261179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.261353 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.262402 (+ 1049us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:41.262407 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:41.265862 (+ 3455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:41.267760 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.267761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.267762 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.268109 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.268317 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.268318 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.268318 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.268374 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.268653 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:41.268654 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:41.268696 (+ 42us) client_negotiation.cc:769] Sending connection context
0618 23:24:41.269146 (+ 450us) client_negotiation.cc:240] Negotiation successful
0618 23:24:41.269262 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":28,"thread_start_us":59,"threads_started":1}
I20260618 23:24:41.269625 15434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.257157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:47673 (local address 127.8.39.188:37721)
0618 23:24:41.257369 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:41.257371 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:41.257378 (+ 7us) server_negotiation.cc:407] Connection header received
0618 23:24:41.258064 (+ 686us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:41.258065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:41.258078 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:41.258141 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:41.258686 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.259708 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.260754 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.261086 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.261180 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.265108 (+ 3928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:41.265110 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:41.265110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:41.265118 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:41.266958 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.268137 (+ 1179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.268138 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.268138 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.268206 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.268395 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.268396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.268397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.268687 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:41.268752 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:41.269030 (+ 278us) server_negotiation.cc:299] Negotiation successful
0618 23:24:41.269526 (+ 496us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196,"thread_start_us":64,"threads_started":1}
I20260618 23:24:41.269923 15429 heartbeater.cc:347] Connected to a master server at 127.8.39.189:44675
I20260618 23:24:41.270035 15429 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:41.270207 15429 heartbeater.cc:511] Master 127.8.39.189:44675 requested a full tablet report, sending...
I20260618 23:24:41.270427 15433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.257133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:37721 (local address 127.8.39.129:47673)
0618 23:24:41.257274 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:41.257284 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:41.257669 (+ 385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:41.258175 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:41.258177 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:41.258485 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:41.258579 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.258586 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.259805 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.259806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.260477 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.260589 (+ 112us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.261239 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.261240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.261349 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.262326 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:41.262336 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:41.264954 (+ 2618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:41.267755 (+ 2801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.267756 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.267756 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.268062 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.268274 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.268275 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.268275 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.268331 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.268778 (+ 447us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:41.268779 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:41.268815 (+ 36us) client_negotiation.cc:769] Sending connection context
0618 23:24:41.268875 (+ 60us) client_negotiation.cc:240] Negotiation successful
0618 23:24:41.268977 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":70,"threads_started":1}
I20260618 23:24:41.270730 15054 ts_manager.cc:194] Registered new tserver with Master: b46d97a84c2b4d88b79434d5e2bf1a16 (127.8.39.129:45209)
I20260618 23:24:41.270787 15428 heartbeater.cc:347] Connected to a master server at 127.8.39.190:41963
I20260618 23:24:41.270834 15428 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:41.270869 15427 heartbeater.cc:347] Connected to a master server at 127.8.39.188:37721
I20260618 23:24:41.270905 15427 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:41.270910 15428 heartbeater.cc:511] Master 127.8.39.190:41963 requested a full tablet report, sending...
I20260618 23:24:41.270975 15427 heartbeater.cc:511] Master 127.8.39.188:37721 requested a full tablet report, sending...
I20260618 23:24:41.271193 14456 ts_manager.cc:194] Re-registered known tserver with Master: b46d97a84c2b4d88b79434d5e2bf1a16 (127.8.39.129:45209)
I20260618 23:24:41.271202 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.130:42141
--local_ip_for_outbound_sockets=127.8.39.130
--tserver_master_addrs=127.8.39.190:41963
--webserver_port=44477
--webserver_interface=127.8.39.130
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.130.keytab
--principal=oryx/127.8.39.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.39.190:41963,127.8.39.189:44675
--tserver_master_addrs=127.8.39.190:41963,127.8.39.189:44675,127.8.39.188:37721 with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:24:41.271569 14456 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.39.129@KRBTEST.COM'} at 127.8.39.129:42565
I20260618 23:24:41.271829 15239 ts_manager.cc:194] Registered new tserver with Master: b46d97a84c2b4d88b79434d5e2bf1a16 (127.8.39.129:45209)
W20260618 23:24:41.351423 15435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:41.351614 15435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:41.351637 15435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:41.352782 15435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:41.352823 15435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:41.352850 15435 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.130
I20260618 23:24:41.354375 15435 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.130.keytab
--principal=oryx/127.8.39.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.130:42141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.39.130
--webserver_port=44477
--tserver_master_addrs=127.8.39.190:41963,127.8.39.189:44675,127.8.39.188:37721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.130
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:41.354625 15435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:41.354862 15435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:41.360463 15441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:41.360486 15440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:41.360464 15443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:41.360796 15435 server_base.cc:1034] running on GCE node
I20260618 23:24:41.361078 15435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:41.361337 15435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:41.362546 15435 hybrid_clock.cc:648] HybridClock initialized: now 1781825081362530 us; error 42 us; skew 500 ppm
Jun 18 23:24:41 dist-test-slave-47qt krb5kdc[14411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825081, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:41.365391 15435 init.cc:377] Logged in from keytab as oryx/127.8.39.130@KRBTEST.COM (short username oryx)
I20260618 23:24:41.365907 15435 webserver.cc:460] Webserver started at http://127.8.39.130:44477/ using document root <none> and password file <none>
I20260618 23:24:41.366139 15435 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:41.366221 15435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:41.367619 15435 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:24:41.368161 15450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:41.368350 15435 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260618 23:24:41.368444 15435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-1/wal
uuid: "a51df27f83cb4e5cab3bcb7bf4d8efa3"
format_stamp: "Formatted at 2026-06-18 23:24:40 on dist-test-slave-47qt"
server_key: "b3bbab56c353eaae033d9aec345aec1b"
server_key_iv: "606c3a84d00884d84b8592a38862aca8"
server_key_version: "encryptionkey@0"
I20260618 23:24:41.368785 15435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:41.382349 15435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:41.384634 15435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:41.384799 15435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:41.385020 15435 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:41.385362 15435 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:41.385419 15435 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:41.385459 15435 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:41.385488 15435 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:41.391463 15435 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.130:42141
I20260618 23:24:41.391502 15563 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.130:42141 every 8 connection(s)
I20260618 23:24:41.391865 15435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
I20260618 23:24:41.396014 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 15435
I20260618 23:24:41.396186 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 14786
Jun 18 23:24:41 dist-test-slave-47qt krb5kdc[14411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825081, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM
Jun 18 23:24:41 dist-test-slave-47qt krb5kdc[14411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825081, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.189@KRBTEST.COM
I20260618 23:24:41.402412 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.131:43739
--local_ip_for_outbound_sockets=127.8.39.131
--tserver_master_addrs=127.8.39.190:41963
--webserver_port=35577
--webserver_interface=127.8.39.131
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.131.keytab
--principal=oryx/127.8.39.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.39.190:41963,127.8.39.189:44675
--tserver_master_addrs=127.8.39.190:41963,127.8.39.189:44675,127.8.39.188:37721 with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:24:41.404356 14496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.392679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:33431 (local address 127.8.39.190:41963)
0618 23:24:41.392920 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:41.392922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:41.393285 (+ 363us) server_negotiation.cc:407] Connection header received
0618 23:24:41.393898 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:41.393899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:41.393912 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:41.393945 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:41.394912 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.395743 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.397362 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.397690 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.397767 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.400577 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:41.400579 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:41.400579 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:41.400588 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:41.402417 (+ 1829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.403153 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.403154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.403155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.403220 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.403510 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.403511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.403511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.403663 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:41.403730 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:41.404125 (+ 395us) server_negotiation.cc:299] Negotiation successful
0618 23:24:41.404223 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217}
I20260618 23:24:41.404745 15105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.392651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:55229 (local address 127.8.39.189:44675)
0618 23:24:41.392702 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:41.392850 (+ 148us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:41.393285 (+ 435us) server_negotiation.cc:407] Connection header received
0618 23:24:41.393898 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:41.393899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:41.393912 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:41.393945 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:41.394912 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.395736 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.399003 (+ 3267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.399326 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.399362 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.401793 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:41.401794 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:41.401795 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:41.401802 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:41.403468 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.403913 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.403914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.403914 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.403965 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.404180 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.404181 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.404181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.404312 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:41.404362 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:41.404583 (+ 221us) server_negotiation.cc:299] Negotiation successful
0618 23:24:41.404660 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Jun 18 23:24:41 dist-test-slave-47qt krb5kdc[14411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825081, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.188@KRBTEST.COM
I20260618 23:24:41.404943 15569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.392597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.8.39.130:33431)
0618 23:24:41.393141 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:41.393194 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:41.393782 (+ 588us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:41.394056 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:41.394061 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:41.394587 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:41.394806 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.394811 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.395875 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.395876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.397267 (+ 1391us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.397270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.397790 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.397791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.397936 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.398527 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:41.398537 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:41.400471 (+ 1934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:41.402751 (+ 2280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.402760 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.402762 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.403081 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.403276 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.403277 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.403277 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.403423 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.403759 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:41.403766 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:41.403937 (+ 171us) client_negotiation.cc:769] Sending connection context
0618 23:24:41.404462 (+ 525us) client_negotiation.cc:240] Negotiation successful
0618 23:24:41.404663 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":2,"thread_start_us":243,"threads_started":1}
I20260618 23:24:41.404950 15568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.392597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:44675 (local address 127.8.39.130:55229)
0618 23:24:41.393162 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:41.393194 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:41.393787 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:41.394056 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:41.394061 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:41.394587 (+ 526us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:41.394806 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.394811 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.397797 (+ 2986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.397797 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.398902 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.398904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.399424 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.399425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.399472 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.399913 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:41.399918 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:41.401678 (+ 1760us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:41.403548 (+ 1870us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.403549 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.403550 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.403829 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.404032 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.404033 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.404034 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.404102 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.404395 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:41.404396 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:41.404438 (+ 42us) client_negotiation.cc:769] Sending connection context
0618 23:24:41.404527 (+ 89us) client_negotiation.cc:240] Negotiation successful
0618 23:24:41.404744 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":151,"thread_start_us":88,"threads_started":1}
I20260618 23:24:41.405442 15565 heartbeater.cc:347] Connected to a master server at 127.8.39.190:41963
I20260618 23:24:41.405570 15565 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:41.405747 15565 heartbeater.cc:511] Master 127.8.39.190:41963 requested a full tablet report, sending...
I20260618 23:24:41.405494 15566 heartbeater.cc:347] Connected to a master server at 127.8.39.189:44675
I20260618 23:24:41.405952 15566 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:41.406028 15566 heartbeater.cc:511] Master 127.8.39.189:44675 requested a full tablet report, sending...
I20260618 23:24:41.406399 15054 ts_manager.cc:194] Registered new tserver with Master: a51df27f83cb4e5cab3bcb7bf4d8efa3 (127.8.39.130:42141)
I20260618 23:24:41.406440 14456 ts_manager.cc:194] Re-registered known tserver with Master: a51df27f83cb4e5cab3bcb7bf4d8efa3 (127.8.39.130:42141)
I20260618 23:24:41.406960 14456 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.39.130@KRBTEST.COM'} at 127.8.39.130:33431
I20260618 23:24:41.408632 15434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.392825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:50295 (local address 127.8.39.188:37721)
0618 23:24:41.392912 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:41.392914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:41.393335 (+ 421us) server_negotiation.cc:407] Connection header received
0618 23:24:41.393883 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:41.393885 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:41.393900 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:41.393986 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:41.394909 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.395629 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.402434 (+ 6805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.402755 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.402801 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.405868 (+ 3067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:41.405870 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:41.405870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:41.405879 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:41.407453 (+ 1574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.407866 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.407867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.407868 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.407928 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.408153 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.408155 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.408155 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.408272 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:41.408322 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:41.408506 (+ 184us) server_negotiation.cc:299] Negotiation successful
0618 23:24:41.408572 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260618 23:24:41.408900 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.392597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:37721 (local address 127.8.39.130:50295)
0618 23:24:41.393257 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:41.393268 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:41.393781 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:41.394417 (+ 636us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:41.394418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:41.394616 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:41.394805 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.394810 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.395741 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.395742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.402317 (+ 6575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.402320 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.402887 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.402888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.402957 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.403531 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:41.403551 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:41.405776 (+ 2225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:41.407554 (+ 1778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.407555 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.407555 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.407763 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.408006 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.408007 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.408007 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.408072 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.408348 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:41.408349 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:41.408397 (+ 48us) client_negotiation.cc:769] Sending connection context
0618 23:24:41.408483 (+ 86us) client_negotiation.cc:240] Negotiation successful
0618 23:24:41.408605 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"mutex_wait_us":293,"thread_start_us":127,"threads_started":1}
I20260618 23:24:41.409521 15564 heartbeater.cc:347] Connected to a master server at 127.8.39.188:37721
I20260618 23:24:41.409581 15564 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:41.409654 15564 heartbeater.cc:511] Master 127.8.39.188:37721 requested a full tablet report, sending...
I20260618 23:24:41.409914 15239 ts_manager.cc:194] Registered new tserver with Master: a51df27f83cb4e5cab3bcb7bf4d8efa3 (127.8.39.130:42141)
W20260618 23:24:41.489868 15571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:41.490061 15571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:41.490085 15571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:41.491210 15571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:41.491251 15571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:41.491278 15571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.131
I20260618 23:24:41.492722 15571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.131.keytab
--principal=oryx/127.8.39.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.131:43739
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.39.131
--webserver_port=35577
--tserver_master_addrs=127.8.39.190:41963,127.8.39.189:44675,127.8.39.188:37721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15571
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.131
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:41.492960 15571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:41.493242 15571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:41.498561 15577 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:41.498538 15576 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:41.498515 15579 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:41.498894 15571 server_base.cc:1034] running on GCE node
I20260618 23:24:41.499135 15571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:41.499400 15571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:41.500610 15571 hybrid_clock.cc:648] HybridClock initialized: now 1781825081500576 us; error 53 us; skew 500 ppm
Jun 18 23:24:41 dist-test-slave-47qt krb5kdc[14411](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825081, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:41.503381 15571 init.cc:377] Logged in from keytab as oryx/127.8.39.131@KRBTEST.COM (short username oryx)
I20260618 23:24:41.503821 15571 webserver.cc:460] Webserver started at http://127.8.39.131:35577/ using document root <none> and password file <none>
I20260618 23:24:41.504057 15571 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:41.504112 15571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:41.505426 15571 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:24:41.506028 15586 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:41.506218 15571 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:24:41.506314 15571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-2/wal
uuid: "e4889f468084485892fd900728fd9312"
format_stamp: "Formatted at 2026-06-18 23:24:40 on dist-test-slave-47qt"
server_key: "0ea8f999a5dddebf3b0477a22a6c7faa"
server_key_iv: "492ec62d2845727ab87301ae570d9c8e"
server_key_version: "encryptionkey@0"
I20260618 23:24:41.506637 15571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:41.530519 15571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:41.532873 15571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:41.533041 15571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:41.533286 15571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:41.533670 15571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:41.533744 15571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:41.533787 15571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:41.533813 15571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:41.539707 15571 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.131:43739
I20260618 23:24:41.539754 15699 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.131:43739 every 8 connection(s)
I20260618 23:24:41.540268 15571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
Jun 18 23:24:41 dist-test-slave-47qt krb5kdc[14411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825081, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM
Jun 18 23:24:41 dist-test-slave-47qt krb5kdc[14411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825081, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.188@KRBTEST.COM
Jun 18 23:24:41 dist-test-slave-47qt krb5kdc[14411](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825081, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.189@KRBTEST.COM
I20260618 23:24:41.550105 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 15571
I20260618 23:24:41.553257 15434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.541412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:42073 (local address 127.8.39.188:37721)
0618 23:24:41.541490 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:41.541492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:41.541615 (+ 123us) server_negotiation.cc:407] Connection header received
0618 23:24:41.542908 (+ 1293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:41.542909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:41.542921 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:41.542954 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:41.543964 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.544744 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.546826 (+ 2082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.547127 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.547195 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.550535 (+ 3340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:41.550536 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:41.550537 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:41.550543 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:41.552184 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.552567 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.552568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.552568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.552619 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.552778 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.552779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.552779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.552884 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:41.552926 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:41.553084 (+ 158us) server_negotiation.cc:299] Negotiation successful
0618 23:24:41.553179 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260618 23:24:41.553757 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.541069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:37721 (local address 127.8.39.131:42073)
0618 23:24:41.541455 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:41.541498 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:41.542779 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:41.543093 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:41.543098 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:41.543572 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:41.543865 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.543870 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.544849 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.544850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.546299 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.546301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.547207 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.547208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.547273 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.547865 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:41.547868 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:41.550433 (+ 2565us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:41.552274 (+ 1841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.552275 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.552275 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.552492 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.552676 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.552677 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.552677 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.552723 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.552952 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:41.552952 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:41.552989 (+ 37us) client_negotiation.cc:769] Sending connection context
0618 23:24:41.553373 (+ 384us) client_negotiation.cc:240] Negotiation successful
0618 23:24:41.553538 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":175,"thread_start_us":138,"threads_started":1}
I20260618 23:24:41.553913 14496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.541375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:44399 (local address 127.8.39.190:41963)
0618 23:24:41.541426 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:41.541429 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:41.541739 (+ 310us) server_negotiation.cc:407] Connection header received
0618 23:24:41.543088 (+ 1349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:41.543089 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:41.543100 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:41.543131 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:41.543993 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.544758 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.546399 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.546682 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.546771 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.549444 (+ 2673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:41.549446 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:41.549446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:41.549453 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:41.551742 (+ 2289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.552184 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.552185 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.552185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.552236 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.552523 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.552524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.552524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.552643 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:41.552700 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:41.553745 (+ 1045us) server_negotiation.cc:299] Negotiation successful
0618 23:24:41.553824 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260618 23:24:41.554270 15700 heartbeater.cc:347] Connected to a master server at 127.8.39.188:37721
I20260618 23:24:41.554363 15700 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:41.554538 15700 heartbeater.cc:511] Master 127.8.39.188:37721 requested a full tablet report, sending...
I20260618 23:24:41.554608 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.541069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.8.39.131:44399)
0618 23:24:41.541690 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:41.541698 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:41.542947 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:41.543253 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:41.543254 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:41.543593 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:41.543865 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.543870 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.544846 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.544847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.546297 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.546300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.546779 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.546780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.546905 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.547448 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:41.547457 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:41.549330 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:41.551827 (+ 2497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.551833 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.551835 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.552111 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.552337 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.552338 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.552338 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.552452 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.552723 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:41.552728 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:41.552871 (+ 143us) client_negotiation.cc:769] Sending connection context
0618 23:24:41.553067 (+ 196us) client_negotiation.cc:240] Negotiation successful
0618 23:24:41.554463 (+ 1396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":599,"mutex_wait_us":82,"spinlock_wait_cycles":88320,"thread_start_us":128,"threads_started":1}
I20260618 23:24:41.554889 15239 ts_manager.cc:194] Registered new tserver with Master: e4889f468084485892fd900728fd9312 (127.8.39.131:43739)
I20260618 23:24:41.557773 15701 heartbeater.cc:347] Connected to a master server at 127.8.39.190:41963
I20260618 23:24:41.557853 15701 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:41.557917 15701 heartbeater.cc:511] Master 127.8.39.190:41963 requested a full tablet report, sending...
I20260618 23:24:41.558161 14456 ts_manager.cc:194] Re-registered known tserver with Master: e4889f468084485892fd900728fd9312 (127.8.39.131:43739)
I20260618 23:24:41.558673 14456 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.39.131@KRBTEST.COM'} at 127.8.39.131:44399
I20260618 23:24:41.562443 15105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.541350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:40845 (local address 127.8.39.189:44675)
0618 23:24:41.541455 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:41.541457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:41.541601 (+ 144us) server_negotiation.cc:407] Connection header received
0618 23:24:41.542897 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:41.542900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:41.542914 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:41.542982 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:41.543997 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.544882 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.547245 (+ 2363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.547562 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.547625 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.557621 (+ 9996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:41.557622 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:41.557622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:41.557630 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:41.560882 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.561352 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.561353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.561353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.561410 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.561639 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.561640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.561641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.561762 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:41.561822 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:41.562272 (+ 450us) server_negotiation.cc:299] Negotiation successful
0618 23:24:41.562348 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260618 23:24:41.563113 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.541070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:44675 (local address 127.8.39.131:40845)
0618 23:24:41.541455 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:41.541498 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:41.542785 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:41.543104 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:41.543106 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:41.543572 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:41.543865 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.543870 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.544990 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.544991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.546299 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.546301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.547624 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.547625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.547688 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.548091 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:41.548095 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:41.557502 (+ 9407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:41.561012 (+ 3510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.561013 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.561013 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.561265 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.561501 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:41.561502 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:41.561503 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:41.561561 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:41.561848 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:41.561849 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:41.561891 (+ 42us) client_negotiation.cc:769] Sending connection context
0618 23:24:41.562744 (+ 853us) client_negotiation.cc:240] Negotiation successful
0618 23:24:41.562876 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":141,"thread_start_us":135,"threads_started":1}
I20260618 23:24:41.563670 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.551151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37722 (local address 127.8.39.188:37721)
0618 23:24:41.551850 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:41.551852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:41.551861 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:24:41.555165 (+ 3304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:41.555166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:41.555176 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:41.555222 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:41.555916 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.556652 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.557617 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.557921 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.557989 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.559448 (+ 1459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:41.559449 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:41.559449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:41.559455 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:41.562023 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.562486 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.562487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.562487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.562610 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.562833 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.562834 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.562835 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.562969 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:41.563077 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:41.563465 (+ 388us) server_negotiation.cc:299] Negotiation successful
0618 23:24:41.563558 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":681,"thread_start_us":52,"threads_started":1}
I20260618 23:24:41.563787 15702 heartbeater.cc:347] Connected to a master server at 127.8.39.189:44675
I20260618 23:24:41.563843 15702 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:41.563915 15702 heartbeater.cc:511] Master 127.8.39.189:44675 requested a full tablet report, sending...
I20260618 23:24:41.564199 15054 ts_manager.cc:194] Registered new tserver with Master: e4889f468084485892fd900728fd9312 (127.8.39.131:43739)
I20260618 23:24:41.566638 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.551626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42392 (local address 127.8.39.189:44675)
0618 23:24:41.552292 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:41.552295 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:41.557440 (+ 5145us) server_negotiation.cc:407] Connection header received
0618 23:24:41.557464 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:41.557465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:41.557478 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:41.557531 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:41.558783 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.559640 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.560822 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.561217 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.561282 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.563137 (+ 1855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:41.563139 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:41.563140 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:41.563146 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:41.565300 (+ 2154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.565648 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.565650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.565650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.565711 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.565857 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.565858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.565858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.565982 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:41.566063 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:41.566502 (+ 439us) server_negotiation.cc:299] Negotiation successful
0618 23:24:41.566579 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":51,"threads_started":1}
I20260618 23:24:41.568550 14496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.552037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46874 (local address 127.8.39.190:41963)
0618 23:24:41.553986 (+ 1949us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:41.553987 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:41.560940 (+ 6953us) server_negotiation.cc:407] Connection header received
0618 23:24:41.561037 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:41.561039 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:41.561051 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:41.561110 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:41.561736 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.562486 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.564114 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.564429 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.564471 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.565941 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:41.565942 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:41.565943 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:41.565948 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:41.567486 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.567826 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.567827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.567828 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.567879 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.568082 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.568083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.568083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.568187 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:41.568287 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:41.568384 (+ 97us) server_negotiation.cc:299] Negotiation successful
0618 23:24:41.568453 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1936,"thread_start_us":88,"threads_started":1}
I20260618 23:24:41.569936 14456 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:24:41.571547 14456 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:24:41.579810 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.575714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:45209 (local address 127.0.0.1:42842)
0618 23:24:41.576069 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:41.576079 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:41.576119 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:41.576519 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:41.576520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:41.576522 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:41.576557 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.576558 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.577524 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.577525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.578621 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.578624 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.579545 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.579546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.579612 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.579616 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:24:41.579677 (+ 61us) client_negotiation.cc:240] Negotiation successful
0618 23:24:41.579722 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":79,"threads_started":1}
I20260618 23:24:41.579807 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.575800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42842 (local address 127.8.39.129:45209)
0618 23:24:41.576358 (+ 558us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:41.576362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:41.576371 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:24:41.576405 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:41.576407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:41.576421 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:41.576503 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:41.576640 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.577438 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.578707 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.579452 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.579527 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.579599 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:41.579691 (+ 92us) server_negotiation.cc:299] Negotiation successful
0618 23:24:41.579728 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":87,"threads_started":1}
I20260618 23:24:41.580143 15014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.575701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:42141 (local address 127.0.0.1:35560)
0618 23:24:41.575918 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:41.575929 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:41.575974 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:41.576587 (+ 613us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:41.576589 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:41.576590 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:41.576625 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.576626 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.577821 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.577822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.579049 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.579052 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.579953 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.579954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.580020 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.580023 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:24:41.580046 (+ 23us) client_negotiation.cc:240] Negotiation successful
0618 23:24:41.580072 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":230}
I20260618 23:24:41.580399 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.576041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42340 (local address 127.8.39.131:43739)
0618 23:24:41.576218 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:41.576221 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:41.576230 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:24:41.576304 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:41.576306 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:41.576322 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:41.576403 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:41.576559 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.577704 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.578958 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.579544 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.579612 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.579687 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:41.580253 (+ 566us) server_negotiation.cc:299] Negotiation successful
0618 23:24:41.580312 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":85,"threads_started":1}
I20260618 23:24:41.580528 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.575781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35560 (local address 127.8.39.130:42141)
0618 23:24:41.575984 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:41.575989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:41.576002 (+ 13us) server_negotiation.cc:407] Connection header received
0618 23:24:41.576044 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:41.576046 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:41.576062 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:41.576598 (+ 536us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:41.576681 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.577740 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.579133 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.579854 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.580303 (+ 449us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.580381 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:41.580405 (+ 24us) server_negotiation.cc:299] Negotiation successful
0618 23:24:41.580445 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":101,"threads_started":1}
I20260618 23:24:41.580571 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.575796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:43739 (local address 127.0.0.1:42340)
0618 23:24:41.576186 (+ 390us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:41.576195 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:41.576241 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:41.576417 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:41.576418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:41.576420 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:41.576459 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.576460 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.577803 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.577805 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.578862 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.578864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.579717 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.579719 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.579767 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.579770 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:24:41.579814 (+ 44us) client_negotiation.cc:240] Negotiation successful
0618 23:24:41.579854 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":371,"mutex_wait_us":122,"thread_start_us":85,"threads_started":1}
I20260618 23:24:41.580607 15361 tablet_service.cc:1467] Processing CreateTablet for tablet d6f40a4cfc364a1a97378ba4e1a34651 (DEFAULT_TABLE table=test-table [id=c9239a549a4c468fba24d6b5b6390247]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:41.580952 15361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6f40a4cfc364a1a97378ba4e1a34651. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:41.581053 15498 tablet_service.cc:1467] Processing CreateTablet for tablet d6f40a4cfc364a1a97378ba4e1a34651 (DEFAULT_TABLE table=test-table [id=c9239a549a4c468fba24d6b5b6390247]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:41.581254 15634 tablet_service.cc:1467] Processing CreateTablet for tablet d6f40a4cfc364a1a97378ba4e1a34651 (DEFAULT_TABLE table=test-table [id=c9239a549a4c468fba24d6b5b6390247]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:41.581377 15498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6f40a4cfc364a1a97378ba4e1a34651. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:41.581591 15634 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d6f40a4cfc364a1a97378ba4e1a34651. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:41.584538 15726 tablet_bootstrap.cc:492] T d6f40a4cfc364a1a97378ba4e1a34651 P e4889f468084485892fd900728fd9312: Bootstrap starting.
I20260618 23:24:41.584764 15727 tablet_bootstrap.cc:492] T d6f40a4cfc364a1a97378ba4e1a34651 P a51df27f83cb4e5cab3bcb7bf4d8efa3: Bootstrap starting.
I20260618 23:24:41.585270 15726 tablet_bootstrap.cc:654] T d6f40a4cfc364a1a97378ba4e1a34651 P e4889f468084485892fd900728fd9312: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:41.585295 15728 tablet_bootstrap.cc:492] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16: Bootstrap starting.
I20260618 23:24:41.585386 15727 tablet_bootstrap.cc:654] T d6f40a4cfc364a1a97378ba4e1a34651 P a51df27f83cb4e5cab3bcb7bf4d8efa3: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:41.585585 15726 log.cc:824] T d6f40a4cfc364a1a97378ba4e1a34651 P e4889f468084485892fd900728fd9312: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:41.585711 15727 log.cc:824] T d6f40a4cfc364a1a97378ba4e1a34651 P a51df27f83cb4e5cab3bcb7bf4d8efa3: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:41.585891 15728 tablet_bootstrap.cc:654] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:41.586211 15728 log.cc:824] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:41.586416 15726 tablet_bootstrap.cc:492] T d6f40a4cfc364a1a97378ba4e1a34651 P e4889f468084485892fd900728fd9312: No bootstrap required, opened a new log
I20260618 23:24:41.586484 15726 ts_tablet_manager.cc:1397] T d6f40a4cfc364a1a97378ba4e1a34651 P e4889f468084485892fd900728fd9312: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:24:41.586972 15728 tablet_bootstrap.cc:492] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16: No bootstrap required, opened a new log
I20260618 23:24:41.587023 15728 ts_tablet_manager.cc:1397] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:24:41.588557 15726 raft_consensus.cc:348] T d6f40a4cfc364a1a97378ba4e1a34651 P e4889f468084485892fd900728fd9312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51df27f83cb4e5cab3bcb7bf4d8efa3" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42141 } } peers { permanent_uuid: "b46d97a84c2b4d88b79434d5e2bf1a16" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45209 } } peers { permanent_uuid: "e4889f468084485892fd900728fd9312" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43739 } }
I20260618 23:24:41.588708 15726 raft_consensus.cc:374] T d6f40a4cfc364a1a97378ba4e1a34651 P e4889f468084485892fd900728fd9312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:41.588750 15726 raft_consensus.cc:729] T d6f40a4cfc364a1a97378ba4e1a34651 P e4889f468084485892fd900728fd9312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4889f468084485892fd900728fd9312, State: Initialized, Role: FOLLOWER
I20260618 23:24:41.588860 15726 consensus_queue.cc:260] T d6f40a4cfc364a1a97378ba4e1a34651 P e4889f468084485892fd900728fd9312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51df27f83cb4e5cab3bcb7bf4d8efa3" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42141 } } peers { permanent_uuid: "b46d97a84c2b4d88b79434d5e2bf1a16" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45209 } } peers { permanent_uuid: "e4889f468084485892fd900728fd9312" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43739 } }
I20260618 23:24:41.589160 15701 heartbeater.cc:503] Master 127.8.39.190:41963 was elected leader, sending a full tablet report...
I20260618 23:24:41.589192 15728 raft_consensus.cc:348] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51df27f83cb4e5cab3bcb7bf4d8efa3" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42141 } } peers { permanent_uuid: "b46d97a84c2b4d88b79434d5e2bf1a16" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45209 } } peers { permanent_uuid: "e4889f468084485892fd900728fd9312" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43739 } }
I20260618 23:24:41.589344 15728 raft_consensus.cc:374] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:41.589377 15728 raft_consensus.cc:729] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b46d97a84c2b4d88b79434d5e2bf1a16, State: Initialized, Role: FOLLOWER
I20260618 23:24:41.589486 15728 consensus_queue.cc:260] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51df27f83cb4e5cab3bcb7bf4d8efa3" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42141 } } peers { permanent_uuid: "b46d97a84c2b4d88b79434d5e2bf1a16" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45209 } } peers { permanent_uuid: "e4889f468084485892fd900728fd9312" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43739 } }
I20260618 23:24:41.589146 15726 ts_tablet_manager.cc:1428] T d6f40a4cfc364a1a97378ba4e1a34651 P e4889f468084485892fd900728fd9312: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:24:41.589764 15728 ts_tablet_manager.cc:1428] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:24:41.590349 15428 heartbeater.cc:503] Master 127.8.39.190:41963 was elected leader, sending a full tablet report...
I20260618 23:24:41.592579 15722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.588290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:37721 (local address 127.0.0.1:37724)
0618 23:24:41.588340 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:41.588351 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:41.588390 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:41.589607 (+ 1217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:41.589608 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:41.589610 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:41.589653 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.589655 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.590546 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.590547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.591576 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.591578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.592380 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.592381 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.592434 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.592437 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:24:41.592494 (+ 57us) client_negotiation.cc:240] Negotiation successful
0618 23:24:41.592529 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260618 23:24:41.592636 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.589407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37724 (local address 127.8.39.188:37721)
0618 23:24:41.589452 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:41.589454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:41.589462 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:24:41.589491 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:41.589492 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:41.589515 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:41.589588 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:41.589729 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.590449 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.591783 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.592289 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.592346 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.592386 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:41.592527 (+ 141us) server_negotiation.cc:299] Negotiation successful
0618 23:24:41.592574 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260618 23:24:41.592790 15727 tablet_bootstrap.cc:492] T d6f40a4cfc364a1a97378ba4e1a34651 P a51df27f83cb4e5cab3bcb7bf4d8efa3: No bootstrap required, opened a new log
I20260618 23:24:41.592861 15727 ts_tablet_manager.cc:1397] T d6f40a4cfc364a1a97378ba4e1a34651 P a51df27f83cb4e5cab3bcb7bf4d8efa3: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260618 23:24:41.592875 15258 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260618 23:24:41.593253 15287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3074d2dc0694e5fa8b0bfa6b3ff985b. Latest consensus state: current_term: 1 leader_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: NON_VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: true } } }
I20260618 23:24:41.593349 15287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [sys.catalog]: This master's current role is: LEARNER
I20260618 23:24:41.594960 15727 raft_consensus.cc:348] T d6f40a4cfc364a1a97378ba4e1a34651 P a51df27f83cb4e5cab3bcb7bf4d8efa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51df27f83cb4e5cab3bcb7bf4d8efa3" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42141 } } peers { permanent_uuid: "b46d97a84c2b4d88b79434d5e2bf1a16" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45209 } } peers { permanent_uuid: "e4889f468084485892fd900728fd9312" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43739 } }
I20260618 23:24:41.595149 15727 raft_consensus.cc:374] T d6f40a4cfc364a1a97378ba4e1a34651 P a51df27f83cb4e5cab3bcb7bf4d8efa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:41.595196 15727 raft_consensus.cc:729] T d6f40a4cfc364a1a97378ba4e1a34651 P a51df27f83cb4e5cab3bcb7bf4d8efa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a51df27f83cb4e5cab3bcb7bf4d8efa3, State: Initialized, Role: FOLLOWER
I20260618 23:24:41.595312 15727 consensus_queue.cc:260] T d6f40a4cfc364a1a97378ba4e1a34651 P a51df27f83cb4e5cab3bcb7bf4d8efa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51df27f83cb4e5cab3bcb7bf4d8efa3" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42141 } } peers { permanent_uuid: "b46d97a84c2b4d88b79434d5e2bf1a16" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45209 } } peers { permanent_uuid: "e4889f468084485892fd900728fd9312" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43739 } }
I20260618 23:24:41.595577 15727 ts_tablet_manager.cc:1428] T d6f40a4cfc364a1a97378ba4e1a34651 P a51df27f83cb4e5cab3bcb7bf4d8efa3: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260618 23:24:41.595892 15565 heartbeater.cc:503] Master 127.8.39.190:41963 was elected leader, sending a full tablet report...
W20260618 23:24:41.643529 15567 tablet.cc:2369] T d6f40a4cfc364a1a97378ba4e1a34651 P a51df27f83cb4e5cab3bcb7bf4d8efa3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 23:24:41.757721 15430 tablet.cc:2369] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 23:24:41.791177 15703 tablet.cc:2369] T d6f40a4cfc364a1a97378ba4e1a34651 P e4889f468084485892fd900728fd9312: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:24:41.807487 15733 raft_consensus.cc:482] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:24:41.807596 15733 raft_consensus.cc:504] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51df27f83cb4e5cab3bcb7bf4d8efa3" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42141 } } peers { permanent_uuid: "b46d97a84c2b4d88b79434d5e2bf1a16" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45209 } } peers { permanent_uuid: "e4889f468084485892fd900728fd9312" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43739 } }
I20260618 23:24:41.808027 15733 leader_election.cc:288] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a51df27f83cb4e5cab3bcb7bf4d8efa3 (127.8.39.130:42141), e4889f468084485892fd900728fd9312 (127.8.39.131:43739)
I20260618 23:24:41.812050 15739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.808359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:43739 (local address 127.8.39.129:54323)
0618 23:24:41.808459 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:41.808468 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:41.808525 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:41.808703 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:41.808704 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:41.808706 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:41.808739 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.808740 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.809624 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.809624 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.810982 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.810985 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.811771 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.811772 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.811837 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.811841 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:24:41.811887 (+ 46us) client_negotiation.cc:240] Negotiation successful
0618 23:24:41.811923 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":123,"thread_start_us":38,"threads_started":1}
I20260618 23:24:41.812069 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.808457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:54323 (local address 127.8.39.131:43739)
0618 23:24:41.808504 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:41.808507 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:41.808515 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:24:41.808619 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:41.808621 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:41.808634 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:41.808687 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:41.808798 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.809550 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.811095 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.811675 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.811755 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.811764 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:41.811948 (+ 184us) server_negotiation.cc:299] Negotiation successful
0618 23:24:41.811999 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260618 23:24:41.812110 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.808293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:60369 (local address 127.8.39.130:42141)
0618 23:24:41.808362 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:41.808364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:41.808395 (+ 31us) server_negotiation.cc:407] Connection header received
0618 23:24:41.808486 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:41.808487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:41.808500 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:41.808589 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:41.808743 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.809528 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.811095 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.811694 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.811755 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.811764 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:41.811986 (+ 222us) server_negotiation.cc:299] Negotiation successful
0618 23:24:41.812034 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260618 23:24:41.812127 15738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.808196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:42141 (local address 127.8.39.129:60369)
0618 23:24:41.808306 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:41.808336 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:41.808417 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:41.808586 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:41.808587 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:41.808589 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:41.808650 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.808651 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.809622 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.809623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.810982 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:41.810985 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.811801 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.811803 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:41.811903 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.811908 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:24:41.811966 (+ 58us) client_negotiation.cc:240] Negotiation successful
0618 23:24:41.812008 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":34,"thread_start_us":57,"threads_started":1}
I20260618 23:24:41.812650 15654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6f40a4cfc364a1a97378ba4e1a34651" candidate_uuid: "b46d97a84c2b4d88b79434d5e2bf1a16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4889f468084485892fd900728fd9312" is_pre_election: true
I20260618 23:24:41.812654 15518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6f40a4cfc364a1a97378ba4e1a34651" candidate_uuid: "b46d97a84c2b4d88b79434d5e2bf1a16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a51df27f83cb4e5cab3bcb7bf4d8efa3" is_pre_election: true
I20260618 23:24:41.812812 15654 raft_consensus.cc:2454] T d6f40a4cfc364a1a97378ba4e1a34651 P e4889f468084485892fd900728fd9312 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b46d97a84c2b4d88b79434d5e2bf1a16 in term 0.
I20260618 23:24:41.812829 15518 raft_consensus.cc:2454] T d6f40a4cfc364a1a97378ba4e1a34651 P a51df27f83cb4e5cab3bcb7bf4d8efa3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b46d97a84c2b4d88b79434d5e2bf1a16 in term 0.
I20260618 23:24:41.813067 15316 leader_election.cc:302] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a51df27f83cb4e5cab3bcb7bf4d8efa3, b46d97a84c2b4d88b79434d5e2bf1a16; no voters:
I20260618 23:24:41.813246 15733 raft_consensus.cc:2781] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:24:41.813308 15733 raft_consensus.cc:482] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:24:41.813325 15733 raft_consensus.cc:3037] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:41.814011 15733 raft_consensus.cc:504] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51df27f83cb4e5cab3bcb7bf4d8efa3" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42141 } } peers { permanent_uuid: "b46d97a84c2b4d88b79434d5e2bf1a16" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45209 } } peers { permanent_uuid: "e4889f468084485892fd900728fd9312" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43739 } }
I20260618 23:24:41.814123 15733 leader_election.cc:288] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16 [CANDIDATE]: Term 1 election: Requested vote from peers a51df27f83cb4e5cab3bcb7bf4d8efa3 (127.8.39.130:42141), e4889f468084485892fd900728fd9312 (127.8.39.131:43739)
I20260618 23:24:41.814343 15518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6f40a4cfc364a1a97378ba4e1a34651" candidate_uuid: "b46d97a84c2b4d88b79434d5e2bf1a16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a51df27f83cb4e5cab3bcb7bf4d8efa3"
I20260618 23:24:41.814425 15518 raft_consensus.cc:3037] T d6f40a4cfc364a1a97378ba4e1a34651 P a51df27f83cb4e5cab3bcb7bf4d8efa3 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:41.814447 15654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d6f40a4cfc364a1a97378ba4e1a34651" candidate_uuid: "b46d97a84c2b4d88b79434d5e2bf1a16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4889f468084485892fd900728fd9312"
I20260618 23:24:41.814553 15654 raft_consensus.cc:3037] T d6f40a4cfc364a1a97378ba4e1a34651 P e4889f468084485892fd900728fd9312 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:41.815249 15654 raft_consensus.cc:2454] T d6f40a4cfc364a1a97378ba4e1a34651 P e4889f468084485892fd900728fd9312 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b46d97a84c2b4d88b79434d5e2bf1a16 in term 1.
I20260618 23:24:41.815250 15518 raft_consensus.cc:2454] T d6f40a4cfc364a1a97378ba4e1a34651 P a51df27f83cb4e5cab3bcb7bf4d8efa3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b46d97a84c2b4d88b79434d5e2bf1a16 in term 1.
I20260618 23:24:41.815459 15316 leader_election.cc:302] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b46d97a84c2b4d88b79434d5e2bf1a16, e4889f468084485892fd900728fd9312; no voters:
I20260618 23:24:41.815589 15733 raft_consensus.cc:2781] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:41.815728 15733 raft_consensus.cc:686] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16 [term 1 LEADER]: Becoming Leader. State: Replica: b46d97a84c2b4d88b79434d5e2bf1a16, State: Running, Role: LEADER
I20260618 23:24:41.815820 15733 consensus_queue.cc:237] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51df27f83cb4e5cab3bcb7bf4d8efa3" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42141 } } peers { permanent_uuid: "b46d97a84c2b4d88b79434d5e2bf1a16" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45209 } } peers { permanent_uuid: "e4889f468084485892fd900728fd9312" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43739 } }
I20260618 23:24:41.816578 14456 catalog_manager.cc:5495] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16 reported cstate change: term changed from 0 to 1, leader changed from <none> to b46d97a84c2b4d88b79434d5e2bf1a16 (127.8.39.129). New cstate: current_term: 1 leader_uuid: "b46d97a84c2b4d88b79434d5e2bf1a16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a51df27f83cb4e5cab3bcb7bf4d8efa3" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42141 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b46d97a84c2b4d88b79434d5e2bf1a16" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4889f468084485892fd900728fd9312" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43739 } health_report { overall_health: UNKNOWN } } }
I20260618 23:24:41.817098 15211 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: NON_VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: true } } }
I20260618 23:24:41.817219 15211 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:41.817921 15211 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b: attempting to promote NON_VOTER 425e3e692fa8473b9a729d973f033081 to VOTER
I20260618 23:24:41.818048 15211 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: false } }
I20260618 23:24:41.818523 15064 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [term 1 FOLLOWER]: Refusing update from remote peer a3074d2dc0694e5fa8b0bfa6b3ff985b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260618 23:24:41.818557 15258 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [term 1 LEARNER]: Refusing update from remote peer a3074d2dc0694e5fa8b0bfa6b3ff985b: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260618 23:24:41.818709 15016 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [LEADER]: Connected to new peer: Peer: permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260618 23:24:41.818786 15211 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20260618 23:24:41.819386 15212 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 425e3e692fa8473b9a729d973f033081 (127.8.39.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: false } } }
I20260618 23:24:41.819504 15064 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 425e3e692fa8473b9a729d973f033081 (127.8.39.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: false } } }
I20260618 23:24:41.820194 15212 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: false } } }
I20260618 23:24:41.820290 15212 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:41.820274 15016 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: false } } }
I20260618 23:24:41.820333 15016 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:41.820344 15744 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: false } } }
I20260618 23:24:41.820436 15744 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:24:41.820466 15258 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 425e3e692fa8473b9a729d973f033081 (127.8.39.188) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: false } } }
I20260618 23:24:41.820209 15211 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: false } } }
I20260618 23:24:41.821288 15211 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:41.821329 15287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: false } } }
I20260618 23:24:41.821415 15287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:24:41.864494 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.861766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42854 (local address 127.8.39.129:45209)
0618 23:24:41.861848 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:41.861850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:41.861858 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:24:41.861927 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:41.861928 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:41.861947 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:41.862026 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:24:41.862190 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.862674 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.863688 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.863887 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.863968 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.864146 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:24:41.864255 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:24:41.864354 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:41.864373 (+ 19us) server_negotiation.cc:299] Negotiation successful
0618 23:24:41.864394 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260618 23:24:41.866394 15654 raft_consensus.cc:1261] T d6f40a4cfc364a1a97378ba4e1a34651 P e4889f468084485892fd900728fd9312 [term 1 FOLLOWER]: Refusing update from remote peer b46d97a84c2b4d88b79434d5e2bf1a16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:24:41.866437 15518 raft_consensus.cc:1261] T d6f40a4cfc364a1a97378ba4e1a34651 P a51df27f83cb4e5cab3bcb7bf4d8efa3 [term 1 FOLLOWER]: Refusing update from remote peer b46d97a84c2b4d88b79434d5e2bf1a16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:24:41.866760 15733 consensus_queue.cc:1035] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a51df27f83cb4e5cab3bcb7bf4d8efa3" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:41.866886 15740 consensus_queue.cc:1035] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4889f468084485892fd900728fd9312" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:41.871692 14456 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:24:41.871778 14456 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:24:41.873555 15747 mvcc.cc:205] Tried to move back new op lower bound from 7298355535322116096 to 7298355535118155776. Current Snapshot: MvccSnapshot[applied={T|T < 7298355535322116096 or (T in {7298355535322116096})}]
I20260618 23:24:41.873986 14456 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3074d2dc0694e5fa8b0bfa6b3ff985b: Sending DeleteTablet for 3 replicas of tablet d6f40a4cfc364a1a97378ba4e1a34651
I20260618 23:24:41.874328 15634 tablet_service.cc:1514] Processing DeleteTablet for tablet d6f40a4cfc364a1a97378ba4e1a34651 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:41 UTC) from {username='oryx', principal='oryx/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:42340
I20260618 23:24:41.874418 15360 tablet_service.cc:1514] Processing DeleteTablet for tablet d6f40a4cfc364a1a97378ba4e1a34651 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:41 UTC) from {username='oryx', principal='oryx/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:42842
I20260618 23:24:41.874423 15498 tablet_service.cc:1514] Processing DeleteTablet for tablet d6f40a4cfc364a1a97378ba4e1a34651 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:41 UTC) from {username='oryx', principal='oryx/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:35560
I20260618 23:24:41.874505 15633 tablet_service.cc:1514] Processing DeleteTablet for tablet d6f40a4cfc364a1a97378ba4e1a34651 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:41 UTC) from {username='oryx', principal='oryx/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:42340
I20260618 23:24:41.874614 15756 tablet_replica.cc:330] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16: stopping tablet replica
I20260618 23:24:41.874650 15361 tablet_service.cc:1514] Processing DeleteTablet for tablet d6f40a4cfc364a1a97378ba4e1a34651 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:41 UTC) from {username='oryx', principal='oryx/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:42842
I20260618 23:24:41.874696 15756 raft_consensus.cc:2229] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16 [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:24:41.874783 15756 raft_consensus.cc:2258] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:41.875273 15757 tablet_replica.cc:330] T d6f40a4cfc364a1a97378ba4e1a34651 P a51df27f83cb4e5cab3bcb7bf4d8efa3: stopping tablet replica
I20260618 23:24:41.875376 15757 raft_consensus.cc:2229] T d6f40a4cfc364a1a97378ba4e1a34651 P a51df27f83cb4e5cab3bcb7bf4d8efa3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:24:41.875470 15757 raft_consensus.cc:2258] T d6f40a4cfc364a1a97378ba4e1a34651 P a51df27f83cb4e5cab3bcb7bf4d8efa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:41.875459 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 14424
I20260618 23:24:41.875511 15756 ts_tablet_manager.cc:1905] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:41.875887 15757 ts_tablet_manager.cc:1905] T d6f40a4cfc364a1a97378ba4e1a34651 P a51df27f83cb4e5cab3bcb7bf4d8efa3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:41.877251 15756 ts_tablet_manager.cc:1918] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:24:41.877314 15756 log.cc:1196] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-0/wal/wals/d6f40a4cfc364a1a97378ba4e1a34651
I20260618 23:24:41.877485 15757 ts_tablet_manager.cc:1918] T d6f40a4cfc364a1a97378ba4e1a34651 P a51df27f83cb4e5cab3bcb7bf4d8efa3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:24:41.877542 15757 log.cc:1196] T d6f40a4cfc364a1a97378ba4e1a34651 P a51df27f83cb4e5cab3bcb7bf4d8efa3: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-1/wal/wals/d6f40a4cfc364a1a97378ba4e1a34651
I20260618 23:24:41.877585 15756 ts_tablet_manager.cc:1939] T d6f40a4cfc364a1a97378ba4e1a34651 P b46d97a84c2b4d88b79434d5e2bf1a16: Deleting consensus metadata
I20260618 23:24:41.877779 15757 ts_tablet_manager.cc:1939] T d6f40a4cfc364a1a97378ba4e1a34651 P a51df27f83cb4e5cab3bcb7bf4d8efa3: Deleting consensus metadata
I20260618 23:24:41.878167 15755 tablet_replica.cc:330] T d6f40a4cfc364a1a97378ba4e1a34651 P e4889f468084485892fd900728fd9312: stopping tablet replica
I20260618 23:24:41.878271 15755 raft_consensus.cc:2229] T d6f40a4cfc364a1a97378ba4e1a34651 P e4889f468084485892fd900728fd9312 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:24:41.878366 15755 raft_consensus.cc:2258] T d6f40a4cfc364a1a97378ba4e1a34651 P e4889f468084485892fd900728fd9312 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:41.878731 15755 ts_tablet_manager.cc:1905] T d6f40a4cfc364a1a97378ba4e1a34651 P e4889f468084485892fd900728fd9312: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:41.880301 15755 ts_tablet_manager.cc:1918] T d6f40a4cfc364a1a97378ba4e1a34651 P e4889f468084485892fd900728fd9312: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:24:41.880360 15755 log.cc:1196] T d6f40a4cfc364a1a97378ba4e1a34651 P e4889f468084485892fd900728fd9312: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-2/wal/wals/d6f40a4cfc364a1a97378ba4e1a34651
I20260618 23:24:41.880596 15755 ts_tablet_manager.cc:1939] T d6f40a4cfc364a1a97378ba4e1a34651 P e4889f468084485892fd900728fd9312: Deleting consensus metadata
W20260618 23:24:41.883224 15590 connection.cc:535] server connection from 127.0.0.1:42340 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260618 23:24:41.883466 15314 connection.cc:535] server connection from 127.0.0.1:42842 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260618 23:24:41.883672 15451 connection.cc:535] server connection from 127.0.0.1:35560 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260618 23:24:41.886294 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.875025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37734 (local address 127.8.39.188:37721)
0618 23:24:41.875067 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:41.875069 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:41.875176 (+ 107us) server_negotiation.cc:407] Connection header received
0618 23:24:41.878098 (+ 2922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:41.878100 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:41.878113 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:41.878145 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:41.879136 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.879860 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.881028 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.881337 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.881372 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.883073 (+ 1701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:41.883074 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:41.883075 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:41.883082 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:41.884966 (+ 1884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.885404 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.885405 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.885405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.885464 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.885684 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.885685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.885686 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.885823 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:41.885897 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:41.886116 (+ 219us) server_negotiation.cc:299] Negotiation successful
0618 23:24:41.886202 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260618 23:24:41.892923 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.885501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42394 (local address 127.8.39.189:44675)
0618 23:24:41.885550 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:41.885552 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:41.885861 (+ 309us) server_negotiation.cc:407] Connection header received
0618 23:24:41.885969 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:41.885970 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:41.885984 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:41.886014 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:41.886975 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.887514 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.888365 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.888671 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.888709 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.890196 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:41.890197 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:41.890198 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:41.890204 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:41.891765 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.892128 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.892129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.892129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.892183 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.892379 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.892387 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.892387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.892515 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:41.892633 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:41.892747 (+ 114us) server_negotiation.cc:299] Negotiation successful
0618 23:24:41.892829 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260618 23:24:41.893539 15434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:41.885790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37742 (local address 127.8.39.188:37721)
0618 23:24:41.885828 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:41.885830 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:41.886363 (+ 533us) server_negotiation.cc:407] Connection header received
0618 23:24:41.886472 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:41.886473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:41.886482 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:41.886508 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:41.887148 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.887873 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.888817 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:41.889143 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:41.889207 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:41.890575 (+ 1368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:41.890576 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:41.890576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:41.890582 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:41.892107 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.892445 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.892446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.892446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.892493 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:41.892702 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:41.892703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:41.892703 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:41.892828 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:41.892907 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:41.893369 (+ 462us) server_negotiation.cc:299] Negotiation successful
0618 23:24:41.893445 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260618 23:24:42.871711 15766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:42.871391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.8.39.130:57685)
0618 23:24:42.871542 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:42.871652 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":84,"threads_started":1}
W20260618 23:24:42.872216 15565 heartbeater.cc:644] Failed to heartbeat to 127.8.39.190:41963 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
I20260618 23:24:42.874783 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:42.874537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.8.39.131:33543)
0618 23:24:42.874643 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:42.874739 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":46,"threads_started":1}
W20260618 23:24:42.875144 15701 heartbeater.cc:644] Failed to heartbeat to 127.8.39.190:41963 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
I20260618 23:24:42.875154 15768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:42.874928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.8.39.129:42775)
0618 23:24:42.875050 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:42.875113 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
W20260618 23:24:42.875347 15428 heartbeater.cc:644] Failed to heartbeat to 127.8.39.190:41963 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
I20260618 23:24:43.374470 15769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a3074d2dc0694e5fa8b0bfa6b3ff985b)
I20260618 23:24:43.374473 15770 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a3074d2dc0694e5fa8b0bfa6b3ff985b)
I20260618 23:24:43.374616 15769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: false } }
I20260618 23:24:43.374639 15770 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: false } }
I20260618 23:24:43.374930 15770 leader_election.cc:288] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3074d2dc0694e5fa8b0bfa6b3ff985b (127.8.39.190:41963), 425e3e692fa8473b9a729d973f033081 (127.8.39.188:37721)
I20260618 23:24:43.375072 15769 leader_election.cc:288] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3074d2dc0694e5fa8b0bfa6b3ff985b (127.8.39.190:41963), 6b7ee65180f14c8cbe67b042b1685a8f (127.8.39.189:44675)
I20260618 23:24:43.375244 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:43.375008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.0.0.1:51488)
0618 23:24:43.375144 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:43.375194 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":50,"threads_started":1}
I20260618 23:24:43.375438 15771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:43.374989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.0.0.1:51472)
0618 23:24:43.375269 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:43.375393 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":46,"threads_started":1}
W20260618 23:24:43.375437 15232 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
W20260618 23:24:43.375644 15038 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
I20260618 23:24:43.376058 15772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:43.375889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.0.0.1:51492)
0618 23:24:43.376006 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:43.376033 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260618 23:24:43.376168 15232 leader_election.cc:334] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a3074d2dc0694e5fa8b0bfa6b3ff985b (127.8.39.190:41963): Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
I20260618 23:24:43.376305 15771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:43.375959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.0.0.1:51506)
0618 23:24:43.376258 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:43.376284 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286}
W20260618 23:24:43.376359 15038 leader_election.cc:334] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a3074d2dc0694e5fa8b0bfa6b3ff985b (127.8.39.190:41963): Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
I20260618 23:24:43.379117 15773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:43.375008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:44675 (local address 127.0.0.1:55854)
0618 23:24:43.375201 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:43.375211 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:43.375266 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:43.375856 (+ 590us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:43.375857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:43.375859 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:43.375936 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:43.375938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:43.376798 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:43.376799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:43.378046 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:43.378048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:43.378840 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:43.378841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:43.378914 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:43.378918 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:24:43.378971 (+ 53us) client_negotiation.cc:240] Negotiation successful
0618 23:24:43.379014 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":4,"thread_start_us":37,"threads_started":1}
I20260618 23:24:43.379123 15774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:43.375132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55854 (local address 127.8.39.189:44675)
0618 23:24:43.375359 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:43.375361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:43.375369 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:24:43.375715 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:43.375716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:43.375727 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:43.375777 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:43.375996 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:43.376710 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:43.378160 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:43.378722 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:43.378826 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:43.378838 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:43.378995 (+ 157us) server_negotiation.cc:299] Negotiation successful
0618 23:24:43.379034 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":40,"threads_started":1}
I20260618 23:24:43.379196 15775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:43.375268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:37721 (local address 127.0.0.1:43734)
0618 23:24:43.375430 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:43.375439 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:43.375489 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:43.375677 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:43.375678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:43.375680 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:43.375741 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:43.375743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:43.376798 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:43.376799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:43.378046 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:43.378049 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:43.378840 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:43.378841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:43.378905 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:43.378909 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:24:43.378971 (+ 62us) client_negotiation.cc:240] Negotiation successful
0618 23:24:43.379014 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":61,"threads_started":1}
I20260618 23:24:43.379134 15776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:43.375450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43734 (local address 127.8.39.188:37721)
0618 23:24:43.375536 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:43.375538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:43.375547 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:24:43.375570 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:43.375571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:43.375581 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:43.375808 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:43.375850 (+ 42us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:43.376710 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:43.378160 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:43.378722 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:43.378811 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:43.378824 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:43.378995 (+ 171us) server_negotiation.cc:299] Negotiation successful
0618 23:24:43.379037 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":71,"thread_start_us":42,"threads_started":1}
I20260618 23:24:43.379563 15064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "425e3e692fa8473b9a729d973f033081" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" is_pre_election: true
I20260618 23:24:43.379595 15258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "425e3e692fa8473b9a729d973f033081" is_pre_election: true
I20260618 23:24:43.379664 15064 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 425e3e692fa8473b9a729d973f033081 in term 1.
I20260618 23:24:43.379707 15258 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6b7ee65180f14c8cbe67b042b1685a8f in term 1.
I20260618 23:24:43.379859 15233 leader_election.cc:302] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 425e3e692fa8473b9a729d973f033081, 6b7ee65180f14c8cbe67b042b1685a8f; no voters: a3074d2dc0694e5fa8b0bfa6b3ff985b
I20260618 23:24:43.379899 15040 leader_election.cc:302] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 425e3e692fa8473b9a729d973f033081, 6b7ee65180f14c8cbe67b042b1685a8f; no voters: a3074d2dc0694e5fa8b0bfa6b3ff985b
I20260618 23:24:43.379978 15769 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260618 23:24:43.380021 15769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a3074d2dc0694e5fa8b0bfa6b3ff985b)
I20260618 23:24:43.380034 15770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260618 23:24:43.380046 15769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [term 1 FOLLOWER]: Advancing to term 2
I20260618 23:24:43.380074 15770 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [term 1 FOLLOWER]: Starting leader election (detected failure of leader a3074d2dc0694e5fa8b0bfa6b3ff985b)
I20260618 23:24:43.380105 15770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [term 1 FOLLOWER]: Advancing to term 2
I20260618 23:24:43.380735 15769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: false } }
I20260618 23:24:43.380739 15770 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: false } }
I20260618 23:24:43.380884 15769 leader_election.cc:288] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [CANDIDATE]: Term 2 election: Requested vote from peers a3074d2dc0694e5fa8b0bfa6b3ff985b (127.8.39.190:41963), 6b7ee65180f14c8cbe67b042b1685a8f (127.8.39.189:44675)
I20260618 23:24:43.380997 15770 leader_election.cc:288] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [CANDIDATE]: Term 2 election: Requested vote from peers a3074d2dc0694e5fa8b0bfa6b3ff985b (127.8.39.190:41963), 425e3e692fa8473b9a729d973f033081 (127.8.39.188:37721)
I20260618 23:24:43.381052 15064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "425e3e692fa8473b9a729d973f033081" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6b7ee65180f14c8cbe67b042b1685a8f"
I20260618 23:24:43.381161 15064 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 425e3e692fa8473b9a729d973f033081 in current term 2: Already voted for candidate 6b7ee65180f14c8cbe67b042b1685a8f in this term.
I20260618 23:24:43.381177 15773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:43.380959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.0.0.1:51522)
0618 23:24:43.381096 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:43.381130 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260618 23:24:43.381340 15775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:43.381181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.0.0.1:51526)
0618 23:24:43.381275 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:43.381313 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260618 23:24:43.381319 15258 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "425e3e692fa8473b9a729d973f033081"
I20260618 23:24:43.381394 15258 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 6b7ee65180f14c8cbe67b042b1685a8f in current term 2: Already voted for candidate 425e3e692fa8473b9a729d973f033081 in this term.
I20260618 23:24:43.381650 15775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:43.381574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.0.0.1:51542)
0618 23:24:43.381604 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:43.381630 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260618 23:24:43.381667 15773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:43.381595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.0.0.1:51550)
0618 23:24:43.381625 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:43.381649 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260618 23:24:43.381747 15038 leader_election.cc:334] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a3074d2dc0694e5fa8b0bfa6b3ff985b (127.8.39.190:41963): Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
I20260618 23:24:43.381793 15038 leader_election.cc:302] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6b7ee65180f14c8cbe67b042b1685a8f; no voters: 425e3e692fa8473b9a729d973f033081, a3074d2dc0694e5fa8b0bfa6b3ff985b
I20260618 23:24:43.381878 15770 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20260618 23:24:43.381923 15232 leader_election.cc:334] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a3074d2dc0694e5fa8b0bfa6b3ff985b (127.8.39.190:41963): Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
I20260618 23:24:43.381968 15232 leader_election.cc:302] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 425e3e692fa8473b9a729d973f033081; no voters: 6b7ee65180f14c8cbe67b042b1685a8f, a3074d2dc0694e5fa8b0bfa6b3ff985b
I20260618 23:24:43.382025 15769 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260618 23:24:43.872748 15779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:43.872535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.8.39.130:54357)
0618 23:24:43.872662 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:43.872705 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":66,"threads_started":1}
I20260618 23:24:43.875666 15780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:43.875458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.8.39.131:47747)
0618 23:24:43.875589 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:43.875626 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
I20260618 23:24:43.875874 15781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:43.875682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.8.39.129:54497)
0618 23:24:43.875800 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:43.875831 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":79,"threads_started":1}
I20260618 23:24:44.873502 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:44.873298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.8.39.130:53611)
0618 23:24:44.873418 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:44.873458 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":58,"threads_started":1}
I20260618 23:24:44.876322 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:44.876135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.8.39.131:56159)
0618 23:24:44.876244 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:44.876277 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":52,"threads_started":1}
I20260618 23:24:44.876487 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:44.876227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.8.39.129:59665)
0618 23:24:44.876368 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:44.876441 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":51,"threads_started":1}
I20260618 23:24:44.984777 15785 raft_consensus.cc:482] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:24:44.984874 15785 raft_consensus.cc:504] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: false } }
I20260618 23:24:44.985077 15785 leader_election.cc:288] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a3074d2dc0694e5fa8b0bfa6b3ff985b (127.8.39.190:41963), 6b7ee65180f14c8cbe67b042b1685a8f (127.8.39.189:44675)
I20260618 23:24:44.985469 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:44.985258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.0.0.1:51768)
0618 23:24:44.985388 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:44.985435 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
I20260618 23:24:44.985450 15064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "425e3e692fa8473b9a729d973f033081" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" is_pre_election: true
I20260618 23:24:44.985586 15064 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 425e3e692fa8473b9a729d973f033081 in term 2.
I20260618 23:24:44.985812 15233 leader_election.cc:302] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 425e3e692fa8473b9a729d973f033081, 6b7ee65180f14c8cbe67b042b1685a8f; no voters:
I20260618 23:24:44.985929 15785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260618 23:24:44.985980 15785 raft_consensus.cc:482] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:24:44.986024 15785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [term 2 FOLLOWER]: Advancing to term 3
I20260618 23:24:44.986063 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:44.985957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.0.0.1:51784)
0618 23:24:44.986010 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:44.986037 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260618 23:24:44.986150 15232 leader_election.cc:334] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a3074d2dc0694e5fa8b0bfa6b3ff985b (127.8.39.190:41963): Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
I20260618 23:24:44.986766 15785 raft_consensus.cc:504] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: false } }
I20260618 23:24:44.986922 15785 leader_election.cc:288] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [CANDIDATE]: Term 3 election: Requested vote from peers a3074d2dc0694e5fa8b0bfa6b3ff985b (127.8.39.190:41963), 6b7ee65180f14c8cbe67b042b1685a8f (127.8.39.189:44675)
I20260618 23:24:44.987085 15064 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "425e3e692fa8473b9a729d973f033081" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "6b7ee65180f14c8cbe67b042b1685a8f"
I20260618 23:24:44.987176 15064 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [term 2 FOLLOWER]: Advancing to term 3
I20260618 23:24:44.987176 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:44.987058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.0.0.1:51790)
0618 23:24:44.987121 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:44.987146 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260618 23:24:44.987490 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:44.987399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.0.0.1:51794)
0618 23:24:44.987448 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:44.987471 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260618 23:24:44.987727 15232 leader_election.cc:334] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a3074d2dc0694e5fa8b0bfa6b3ff985b (127.8.39.190:41963): Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
I20260618 23:24:44.988085 15064 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 425e3e692fa8473b9a729d973f033081 in term 3.
I20260618 23:24:44.988274 15233 leader_election.cc:302] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 425e3e692fa8473b9a729d973f033081, 6b7ee65180f14c8cbe67b042b1685a8f; no voters: a3074d2dc0694e5fa8b0bfa6b3ff985b
I20260618 23:24:44.988366 15785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [term 3 FOLLOWER]: Leader election won for term 3
I20260618 23:24:44.988471 15785 raft_consensus.cc:686] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [term 3 LEADER]: Becoming Leader. State: Replica: 425e3e692fa8473b9a729d973f033081, State: Running, Role: LEADER
I20260618 23:24:44.988543 15785 consensus_queue.cc:237] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: false } }
I20260618 23:24:44.989270 15788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 425e3e692fa8473b9a729d973f033081. Latest consensus state: current_term: 3 leader_uuid: "425e3e692fa8473b9a729d973f033081" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: false } } }
I20260618 23:24:44.989382 15788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:44.989596 15790 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:24:44.990144 15790 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:24:44.990239 15790 catalog_manager.cc:1259] Loaded cluster ID: 05a830b28cdd4f759c72bf6b6c1a803a
I20260618 23:24:44.990272 15790 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:24:44.990451 15790 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:24:44.990551 15790 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081: Loaded TSK: 0
I20260618 23:24:44.990700 15790 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:24:45.060130 15240 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37742:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:24:45.061802 15240 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:24:45.062928 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:45.062796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.0.0.1:51822)
0618 23:24:45.062867 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:45.062902 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260618 23:24:45.062955 15064 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [term 3 FOLLOWER]: Refusing update from remote peer 425e3e692fa8473b9a729d973f033081: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260618 23:24:45.063174 15788 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260618 23:24:45.063325 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:45.063223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.0.0.1:51836)
0618 23:24:45.063272 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:45.063297 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260618 23:24:45.063421 15232 consensus_peers.cc:487] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 -> Peer a3074d2dc0694e5fa8b0bfa6b3ff985b (127.8.39.190:41963): Couldn't send request to peer a3074d2dc0694e5fa8b0bfa6b3ff985b. Status: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260618 23:24:45.063812 15792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 425e3e692fa8473b9a729d973f033081. Latest consensus state: current_term: 3 leader_uuid: "425e3e692fa8473b9a729d973f033081" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: false } } }
I20260618 23:24:45.063926 15792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:24:45.064381 15785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "425e3e692fa8473b9a729d973f033081" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: false } } }
I20260618 23:24:45.064582 15785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081 [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:45.064608 15792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "425e3e692fa8473b9a729d973f033081" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "a3074d2dc0694e5fa8b0bfa6b3ff985b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41963 } } peers { permanent_uuid: "6b7ee65180f14c8cbe67b042b1685a8f" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44675 } attrs { promote: false } } peers { permanent_uuid: "425e3e692fa8473b9a729d973f033081" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 37721 } attrs { promote: false } } }
I20260618 23:24:45.064677 15792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6b7ee65180f14c8cbe67b042b1685a8f [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:24:45.069510 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:45.066104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38168 (local address 127.8.39.131:43739)
0618 23:24:45.066217 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:45.066220 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:45.066228 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:24:45.066255 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:45.066256 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:45.066266 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:45.066542 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:45.066590 (+ 48us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:45.067361 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:45.068692 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:45.069102 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:45.069225 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:45.069238 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:45.069397 (+ 159us) server_negotiation.cc:299] Negotiation successful
0618 23:24:45.069436 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":49,"threads_started":1}
I20260618 23:24:45.069547 15786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:45.066031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:43739 (local address 127.0.0.1:38168)
0618 23:24:45.066098 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:45.066110 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:45.066176 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:45.066373 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:45.066374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:45.066376 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:45.066436 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:45.066438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:45.067451 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:45.067452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:45.068559 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:45.068561 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:45.069236 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:45.069237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:45.069302 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:45.069311 (+ 9us) client_negotiation.cc:769] Sending connection context
0618 23:24:45.069389 (+ 78us) client_negotiation.cc:240] Negotiation successful
0618 23:24:45.069431 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":23}
I20260618 23:24:45.069561 15797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:45.066071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:45209 (local address 127.0.0.1:57496)
0618 23:24:45.066262 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:45.066270 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:45.066305 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:45.066557 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:45.066557 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:45.066560 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:45.066593 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:45.066594 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:45.067336 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:45.067337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:45.068419 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:45.068420 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:45.069321 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:45.069322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:45.069381 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:45.069385 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:24:45.069428 (+ 43us) client_negotiation.cc:240] Negotiation successful
0618 23:24:45.069465 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":66,"threads_started":1}
I20260618 23:24:45.069820 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:45.066273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57496 (local address 127.8.39.129:45209)
0618 23:24:45.066386 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:45.066388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:45.066396 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:24:45.066423 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:45.066424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:45.066438 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:45.066551 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:45.066669 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:45.067216 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:45.068613 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:45.069216 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:45.069279 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:45.069291 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:45.069703 (+ 412us) server_negotiation.cc:299] Negotiation successful
0618 23:24:45.069750 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":53,"threads_started":1}
I20260618 23:24:45.070088 15361 tablet_service.cc:1467] Processing CreateTablet for tablet 5f9dae246cac497e94772de2d5a03372 (DEFAULT_TABLE table=test-table [id=52ed044638664ec08ddd0532d753bb9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:45.070261 15361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f9dae246cac497e94772de2d5a03372. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:45.070283 15633 tablet_service.cc:1467] Processing CreateTablet for tablet 5f9dae246cac497e94772de2d5a03372 (DEFAULT_TABLE table=test-table [id=52ed044638664ec08ddd0532d753bb9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:45.070430 15633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f9dae246cac497e94772de2d5a03372. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:45.071604 15802 tablet_bootstrap.cc:492] T 5f9dae246cac497e94772de2d5a03372 P b46d97a84c2b4d88b79434d5e2bf1a16: Bootstrap starting.
I20260618 23:24:45.071924 15803 tablet_bootstrap.cc:492] T 5f9dae246cac497e94772de2d5a03372 P e4889f468084485892fd900728fd9312: Bootstrap starting.
I20260618 23:24:45.071995 15802 tablet_bootstrap.cc:654] T 5f9dae246cac497e94772de2d5a03372 P b46d97a84c2b4d88b79434d5e2bf1a16: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:45.072283 15803 tablet_bootstrap.cc:654] T 5f9dae246cac497e94772de2d5a03372 P e4889f468084485892fd900728fd9312: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:45.072316 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:45.066129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:42141 (local address 127.0.0.1:42970)
0618 23:24:45.066410 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:45.066418 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:45.066473 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:45.066795 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:45.066796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:45.066798 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:45.066835 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:45.066836 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:45.067677 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:45.067678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:45.069657 (+ 1979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:45.069659 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:45.072041 (+ 2382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:45.072043 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:45.072117 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:45.072122 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:24:45.072188 (+ 66us) client_negotiation.cc:240] Negotiation successful
0618 23:24:45.072237 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":76,"threads_started":1}
I20260618 23:24:45.072345 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:45.066296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42970 (local address 127.8.39.130:42141)
0618 23:24:45.066425 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:45.066427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:45.066462 (+ 35us) server_negotiation.cc:407] Connection header received
0618 23:24:45.066678 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:45.066679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:45.066690 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:45.066752 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:45.066899 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:45.067593 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:45.069726 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:45.071937 (+ 2211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:45.072015 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:45.072029 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:45.072198 (+ 169us) server_negotiation.cc:299] Negotiation successful
0618 23:24:45.072250 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":65,"threads_started":1}
I20260618 23:24:45.072611 15802 tablet_bootstrap.cc:492] T 5f9dae246cac497e94772de2d5a03372 P b46d97a84c2b4d88b79434d5e2bf1a16: No bootstrap required, opened a new log
I20260618 23:24:45.072666 15802 ts_tablet_manager.cc:1397] T 5f9dae246cac497e94772de2d5a03372 P b46d97a84c2b4d88b79434d5e2bf1a16: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:45.072777 15802 raft_consensus.cc:348] T 5f9dae246cac497e94772de2d5a03372 P b46d97a84c2b4d88b79434d5e2bf1a16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4889f468084485892fd900728fd9312" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43739 } } peers { permanent_uuid: "b46d97a84c2b4d88b79434d5e2bf1a16" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45209 } } peers { permanent_uuid: "a51df27f83cb4e5cab3bcb7bf4d8efa3" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42141 } }
I20260618 23:24:45.072868 15802 raft_consensus.cc:374] T 5f9dae246cac497e94772de2d5a03372 P b46d97a84c2b4d88b79434d5e2bf1a16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:45.072898 15802 raft_consensus.cc:729] T 5f9dae246cac497e94772de2d5a03372 P b46d97a84c2b4d88b79434d5e2bf1a16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b46d97a84c2b4d88b79434d5e2bf1a16, State: Initialized, Role: FOLLOWER
I20260618 23:24:45.072942 15802 consensus_queue.cc:260] T 5f9dae246cac497e94772de2d5a03372 P b46d97a84c2b4d88b79434d5e2bf1a16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4889f468084485892fd900728fd9312" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43739 } } peers { permanent_uuid: "b46d97a84c2b4d88b79434d5e2bf1a16" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45209 } } peers { permanent_uuid: "a51df27f83cb4e5cab3bcb7bf4d8efa3" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42141 } }
I20260618 23:24:45.073016 15803 tablet_bootstrap.cc:492] T 5f9dae246cac497e94772de2d5a03372 P e4889f468084485892fd900728fd9312: No bootstrap required, opened a new log
I20260618 23:24:45.073063 15803 ts_tablet_manager.cc:1397] T 5f9dae246cac497e94772de2d5a03372 P e4889f468084485892fd900728fd9312: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:45.073215 15498 tablet_service.cc:1467] Processing CreateTablet for tablet 5f9dae246cac497e94772de2d5a03372 (DEFAULT_TABLE table=test-table [id=52ed044638664ec08ddd0532d753bb9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:45.073330 15498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5f9dae246cac497e94772de2d5a03372. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:45.073426 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:45.073312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.8.39.129:55595)
0618 23:24:45.073368 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:45.073400 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260618 23:24:45.073876 15803 raft_consensus.cc:348] T 5f9dae246cac497e94772de2d5a03372 P e4889f468084485892fd900728fd9312 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4889f468084485892fd900728fd9312" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43739 } } peers { permanent_uuid: "b46d97a84c2b4d88b79434d5e2bf1a16" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45209 } } peers { permanent_uuid: "a51df27f83cb4e5cab3bcb7bf4d8efa3" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42141 } }
I20260618 23:24:45.073961 15803 raft_consensus.cc:374] T 5f9dae246cac497e94772de2d5a03372 P e4889f468084485892fd900728fd9312 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:45.073987 15803 raft_consensus.cc:729] T 5f9dae246cac497e94772de2d5a03372 P e4889f468084485892fd900728fd9312 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4889f468084485892fd900728fd9312, State: Initialized, Role: FOLLOWER
I20260618 23:24:45.074045 15803 consensus_queue.cc:260] T 5f9dae246cac497e94772de2d5a03372 P e4889f468084485892fd900728fd9312 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4889f468084485892fd900728fd9312" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43739 } } peers { permanent_uuid: "b46d97a84c2b4d88b79434d5e2bf1a16" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45209 } } peers { permanent_uuid: "a51df27f83cb4e5cab3bcb7bf4d8efa3" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42141 } }
I20260618 23:24:45.074299 15803 ts_tablet_manager.cc:1428] T 5f9dae246cac497e94772de2d5a03372 P e4889f468084485892fd900728fd9312: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:45.074609 15700 heartbeater.cc:503] Master 127.8.39.188:37721 was elected leader, sending a full tablet report...
I20260618 23:24:45.074808 15808 tablet_bootstrap.cc:492] T 5f9dae246cac497e94772de2d5a03372 P a51df27f83cb4e5cab3bcb7bf4d8efa3: Bootstrap starting.
I20260618 23:24:45.074875 15802 ts_tablet_manager.cc:1428] T 5f9dae246cac497e94772de2d5a03372 P b46d97a84c2b4d88b79434d5e2bf1a16: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:24:45.074934 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:45.074841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.8.39.131:42437)
0618 23:24:45.074882 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:45.074906 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260618 23:24:45.074951 15427 heartbeater.cc:503] Master 127.8.39.188:37721 was elected leader, sending a full tablet report...
I20260618 23:24:45.075179 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:45.075054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.8.39.129:35251)
0618 23:24:45.075128 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:45.075156 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260618 23:24:45.075361 15808 tablet_bootstrap.cc:654] T 5f9dae246cac497e94772de2d5a03372 P a51df27f83cb4e5cab3bcb7bf4d8efa3: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:45.075960 15808 tablet_bootstrap.cc:492] T 5f9dae246cac497e94772de2d5a03372 P a51df27f83cb4e5cab3bcb7bf4d8efa3: No bootstrap required, opened a new log
I20260618 23:24:45.076002 15808 ts_tablet_manager.cc:1397] T 5f9dae246cac497e94772de2d5a03372 P a51df27f83cb4e5cab3bcb7bf4d8efa3: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:45.076112 15808 raft_consensus.cc:348] T 5f9dae246cac497e94772de2d5a03372 P a51df27f83cb4e5cab3bcb7bf4d8efa3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4889f468084485892fd900728fd9312" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43739 } } peers { permanent_uuid: "b46d97a84c2b4d88b79434d5e2bf1a16" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45209 } } peers { permanent_uuid: "a51df27f83cb4e5cab3bcb7bf4d8efa3" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42141 } }
I20260618 23:24:45.076161 15808 raft_consensus.cc:374] T 5f9dae246cac497e94772de2d5a03372 P a51df27f83cb4e5cab3bcb7bf4d8efa3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:45.076174 15808 raft_consensus.cc:729] T 5f9dae246cac497e94772de2d5a03372 P a51df27f83cb4e5cab3bcb7bf4d8efa3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a51df27f83cb4e5cab3bcb7bf4d8efa3, State: Initialized, Role: FOLLOWER
I20260618 23:24:45.076236 15808 consensus_queue.cc:260] T 5f9dae246cac497e94772de2d5a03372 P a51df27f83cb4e5cab3bcb7bf4d8efa3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4889f468084485892fd900728fd9312" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43739 } } peers { permanent_uuid: "b46d97a84c2b4d88b79434d5e2bf1a16" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45209 } } peers { permanent_uuid: "a51df27f83cb4e5cab3bcb7bf4d8efa3" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42141 } }
I20260618 23:24:45.076431 15808 ts_tablet_manager.cc:1428] T 5f9dae246cac497e94772de2d5a03372 P a51df27f83cb4e5cab3bcb7bf4d8efa3: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260618 23:24:45.076659 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:45.076553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.8.39.130:52069)
0618 23:24:45.076604 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:45.076632 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260618 23:24:45.076834 15564 heartbeater.cc:503] Master 127.8.39.188:37721 was elected leader, sending a full tablet report...
I20260618 23:24:45.076936 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:45.076849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.8.39.130:46169)
0618 23:24:45.076882 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:45.076912 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260618 23:24:45.195933 15810 raft_consensus.cc:482] T 5f9dae246cac497e94772de2d5a03372 P a51df27f83cb4e5cab3bcb7bf4d8efa3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:24:45.196064 15810 raft_consensus.cc:504] T 5f9dae246cac497e94772de2d5a03372 P a51df27f83cb4e5cab3bcb7bf4d8efa3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4889f468084485892fd900728fd9312" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43739 } } peers { permanent_uuid: "b46d97a84c2b4d88b79434d5e2bf1a16" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45209 } } peers { permanent_uuid: "a51df27f83cb4e5cab3bcb7bf4d8efa3" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42141 } }
I20260618 23:24:45.196451 15810 leader_election.cc:288] T 5f9dae246cac497e94772de2d5a03372 P a51df27f83cb4e5cab3bcb7bf4d8efa3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e4889f468084485892fd900728fd9312 (127.8.39.131:43739), b46d97a84c2b4d88b79434d5e2bf1a16 (127.8.39.129:45209)
I20260618 23:24:45.199545 15798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:45.196796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:41277 (local address 127.8.39.131:43739)
0618 23:24:45.196836 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:45.196838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:45.196846 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:24:45.196877 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:45.196877 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:45.196889 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:45.196949 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:45.197228 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:45.197719 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:45.198826 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:45.199188 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:45.199275 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:45.199285 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:45.199452 (+ 167us) server_negotiation.cc:299] Negotiation successful
0618 23:24:45.199488 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260618 23:24:45.199545 15782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:45.196653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:43739 (local address 127.8.39.130:41277)
0618 23:24:45.196706 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:45.196721 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:45.196781 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:45.196985 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:45.196986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:45.196989 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:45.197075 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:45.197078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:45.197811 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:45.197812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:45.198732 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:45.198735 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:45.199286 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:45.199287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:45.199366 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:45.199370 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:24:45.199434 (+ 64us) client_negotiation.cc:240] Negotiation successful
0618 23:24:45.199472 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260618 23:24:45.199658 15800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:45.196732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:48555 (local address 127.8.39.129:45209)
0618 23:24:45.196778 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:45.196780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:45.196963 (+ 183us) server_negotiation.cc:407] Connection header received
0618 23:24:45.196994 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:45.196995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:45.197009 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:45.197075 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:45.197291 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:45.197827 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:45.198757 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:45.199244 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:45.199320 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:45.199333 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:45.199529 (+ 196us) server_negotiation.cc:299] Negotiation successful
0618 23:24:45.199572 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260618 23:24:45.199818 15654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f9dae246cac497e94772de2d5a03372" candidate_uuid: "a51df27f83cb4e5cab3bcb7bf4d8efa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4889f468084485892fd900728fd9312" is_pre_election: true
I20260618 23:24:45.199879 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:45.196664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:45209 (local address 127.8.39.130:48555)
0618 23:24:45.196843 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:45.196855 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:45.196919 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:45.197090 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:45.197091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:45.197093 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:45.197200 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:45.197202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:45.197918 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:45.197919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:45.198656 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:45.198658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:45.199362 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:45.199364 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:45.199435 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:45.199438 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:24:45.199521 (+ 83us) client_negotiation.cc:240] Negotiation successful
0618 23:24:45.199568 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":3,"thread_start_us":112,"threads_started":1}
I20260618 23:24:45.199918 15654 raft_consensus.cc:2454] T 5f9dae246cac497e94772de2d5a03372 P e4889f468084485892fd900728fd9312 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a51df27f83cb4e5cab3bcb7bf4d8efa3 in term 0.
I20260618 23:24:45.200115 15453 leader_election.cc:302] T 5f9dae246cac497e94772de2d5a03372 P a51df27f83cb4e5cab3bcb7bf4d8efa3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a51df27f83cb4e5cab3bcb7bf4d8efa3, e4889f468084485892fd900728fd9312; no voters:
I20260618 23:24:45.200157 15381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f9dae246cac497e94772de2d5a03372" candidate_uuid: "a51df27f83cb4e5cab3bcb7bf4d8efa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b46d97a84c2b4d88b79434d5e2bf1a16" is_pre_election: true
I20260618 23:24:45.200237 15810 raft_consensus.cc:2781] T 5f9dae246cac497e94772de2d5a03372 P a51df27f83cb4e5cab3bcb7bf4d8efa3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:24:45.200269 15810 raft_consensus.cc:482] T 5f9dae246cac497e94772de2d5a03372 P a51df27f83cb4e5cab3bcb7bf4d8efa3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:24:45.200299 15381 raft_consensus.cc:2454] T 5f9dae246cac497e94772de2d5a03372 P b46d97a84c2b4d88b79434d5e2bf1a16 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a51df27f83cb4e5cab3bcb7bf4d8efa3 in term 0.
I20260618 23:24:45.200323 15810 raft_consensus.cc:3037] T 5f9dae246cac497e94772de2d5a03372 P a51df27f83cb4e5cab3bcb7bf4d8efa3 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:45.200986 15810 raft_consensus.cc:504] T 5f9dae246cac497e94772de2d5a03372 P a51df27f83cb4e5cab3bcb7bf4d8efa3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4889f468084485892fd900728fd9312" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43739 } } peers { permanent_uuid: "b46d97a84c2b4d88b79434d5e2bf1a16" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45209 } } peers { permanent_uuid: "a51df27f83cb4e5cab3bcb7bf4d8efa3" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42141 } }
I20260618 23:24:45.201134 15810 leader_election.cc:288] T 5f9dae246cac497e94772de2d5a03372 P a51df27f83cb4e5cab3bcb7bf4d8efa3 [CANDIDATE]: Term 1 election: Requested vote from peers e4889f468084485892fd900728fd9312 (127.8.39.131:43739), b46d97a84c2b4d88b79434d5e2bf1a16 (127.8.39.129:45209)
I20260618 23:24:45.201310 15654 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f9dae246cac497e94772de2d5a03372" candidate_uuid: "a51df27f83cb4e5cab3bcb7bf4d8efa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4889f468084485892fd900728fd9312"
I20260618 23:24:45.201371 15654 raft_consensus.cc:3037] T 5f9dae246cac497e94772de2d5a03372 P e4889f468084485892fd900728fd9312 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:45.201303 15381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5f9dae246cac497e94772de2d5a03372" candidate_uuid: "a51df27f83cb4e5cab3bcb7bf4d8efa3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b46d97a84c2b4d88b79434d5e2bf1a16"
I20260618 23:24:45.201406 15381 raft_consensus.cc:3037] T 5f9dae246cac497e94772de2d5a03372 P b46d97a84c2b4d88b79434d5e2bf1a16 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:45.202018 15654 raft_consensus.cc:2454] T 5f9dae246cac497e94772de2d5a03372 P e4889f468084485892fd900728fd9312 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a51df27f83cb4e5cab3bcb7bf4d8efa3 in term 1.
I20260618 23:24:45.202097 15381 raft_consensus.cc:2454] T 5f9dae246cac497e94772de2d5a03372 P b46d97a84c2b4d88b79434d5e2bf1a16 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a51df27f83cb4e5cab3bcb7bf4d8efa3 in term 1.
I20260618 23:24:45.202181 15453 leader_election.cc:302] T 5f9dae246cac497e94772de2d5a03372 P a51df27f83cb4e5cab3bcb7bf4d8efa3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a51df27f83cb4e5cab3bcb7bf4d8efa3, e4889f468084485892fd900728fd9312; no voters:
I20260618 23:24:45.202287 15810 raft_consensus.cc:2781] T 5f9dae246cac497e94772de2d5a03372 P a51df27f83cb4e5cab3bcb7bf4d8efa3 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:45.202445 15810 raft_consensus.cc:686] T 5f9dae246cac497e94772de2d5a03372 P a51df27f83cb4e5cab3bcb7bf4d8efa3 [term 1 LEADER]: Becoming Leader. State: Replica: a51df27f83cb4e5cab3bcb7bf4d8efa3, State: Running, Role: LEADER
I20260618 23:24:45.202631 15810 consensus_queue.cc:237] T 5f9dae246cac497e94772de2d5a03372 P a51df27f83cb4e5cab3bcb7bf4d8efa3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4889f468084485892fd900728fd9312" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43739 } } peers { permanent_uuid: "b46d97a84c2b4d88b79434d5e2bf1a16" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45209 } } peers { permanent_uuid: "a51df27f83cb4e5cab3bcb7bf4d8efa3" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42141 } }
I20260618 23:24:45.202912 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:45.202674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.8.39.130:47383)
0618 23:24:45.202852 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:45.202886 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260618 23:24:45.203500 15240 catalog_manager.cc:5495] T 5f9dae246cac497e94772de2d5a03372 P a51df27f83cb4e5cab3bcb7bf4d8efa3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a51df27f83cb4e5cab3bcb7bf4d8efa3 (127.8.39.130). New cstate: current_term: 1 leader_uuid: "a51df27f83cb4e5cab3bcb7bf4d8efa3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e4889f468084485892fd900728fd9312" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43739 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b46d97a84c2b4d88b79434d5e2bf1a16" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a51df27f83cb4e5cab3bcb7bf4d8efa3" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42141 } health_report { overall_health: HEALTHY } } }
I20260618 23:24:45.214803 15801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:45.212417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42984 (local address 127.8.39.130:42141)
0618 23:24:45.212472 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:45.212474 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:45.212482 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:24:45.212512 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:45.212513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:45.212538 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:45.212603 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:24:45.212753 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:45.213273 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:45.214018 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:45.214225 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:45.214299 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:45.214471 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:24:45.214589 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:24:45.214651 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:45.214732 (+ 81us) server_negotiation.cc:299] Negotiation successful
0618 23:24:45.214749 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260618 23:24:45.215965 15381 raft_consensus.cc:1261] T 5f9dae246cac497e94772de2d5a03372 P b46d97a84c2b4d88b79434d5e2bf1a16 [term 1 FOLLOWER]: Refusing update from remote peer a51df27f83cb4e5cab3bcb7bf4d8efa3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:24:45.215966 15654 raft_consensus.cc:1261] T 5f9dae246cac497e94772de2d5a03372 P e4889f468084485892fd900728fd9312 [term 1 FOLLOWER]: Refusing update from remote peer a51df27f83cb4e5cab3bcb7bf4d8efa3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:24:45.216328 15812 consensus_queue.cc:1035] T 5f9dae246cac497e94772de2d5a03372 P a51df27f83cb4e5cab3bcb7bf4d8efa3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4889f468084485892fd900728fd9312" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:45.216576 15810 consensus_queue.cc:1035] T 5f9dae246cac497e94772de2d5a03372 P a51df27f83cb4e5cab3bcb7bf4d8efa3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b46d97a84c2b4d88b79434d5e2bf1a16" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:45.217558 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:45.217465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.8.39.129:36263)
0618 23:24:45.217502 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:45.217529 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260618 23:24:45.217943 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:45.217850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.8.39.131:47495)
0618 23:24:45.217892 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:45.217916 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260618 23:24:45.218945 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:45.218796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.8.39.130:36823)
0618 23:24:45.218887 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:45.218913 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260618 23:24:45.219085 15784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:45.218995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.8.39.129:59191)
0618 23:24:45.219029 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:45.219054 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260618 23:24:45.219511 15783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:45.219313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41963 (local address 127.8.39.131:46157)
0618 23:24:45.219447 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:45.219487 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260618 23:24:45.219805 15817 mvcc.cc:205] Tried to move back new op lower bound from 7298355549042335744 to 7298355548990783488. Current Snapshot: MvccSnapshot[applied={T|T < 7298355549042335744}]
I20260618 23:24:45.221024 15240 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:24:45.221122 15240 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37742:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:24:45.222355 15240 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 425e3e692fa8473b9a729d973f033081: Sending DeleteTablet for 3 replicas of tablet 5f9dae246cac497e94772de2d5a03372
I20260618 23:24:45.222944 15361 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f9dae246cac497e94772de2d5a03372 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:45 UTC) from {username='oryx', principal='oryx/127.8.39.188@KRBTEST.COM'} at 127.0.0.1:57496
I20260618 23:24:45.222922 15633 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f9dae246cac497e94772de2d5a03372 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:45 UTC) from {username='oryx', principal='oryx/127.8.39.188@KRBTEST.COM'} at 127.0.0.1:38168
I20260618 23:24:45.223066 15498 tablet_service.cc:1514] Processing DeleteTablet for tablet 5f9dae246cac497e94772de2d5a03372 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:45 UTC) from {username='oryx', principal='oryx/127.8.39.188@KRBTEST.COM'} at 127.0.0.1:42970
I20260618 23:24:45.223428 15827 tablet_replica.cc:330] T 5f9dae246cac497e94772de2d5a03372 P a51df27f83cb4e5cab3bcb7bf4d8efa3: stopping tablet replica
I20260618 23:24:45.223423 15825 tablet_replica.cc:330] T 5f9dae246cac497e94772de2d5a03372 P b46d97a84c2b4d88b79434d5e2bf1a16: stopping tablet replica
I20260618 23:24:45.223492 15825 raft_consensus.cc:2229] T 5f9dae246cac497e94772de2d5a03372 P b46d97a84c2b4d88b79434d5e2bf1a16 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:24:45.223548 15827 raft_consensus.cc:2229] T 5f9dae246cac497e94772de2d5a03372 P a51df27f83cb4e5cab3bcb7bf4d8efa3 [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:24:45.223610 15825 raft_consensus.cc:2258] T 5f9dae246cac497e94772de2d5a03372 P b46d97a84c2b4d88b79434d5e2bf1a16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:45.223690 15827 raft_consensus.cc:2258] T 5f9dae246cac497e94772de2d5a03372 P a51df27f83cb4e5cab3bcb7bf4d8efa3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:45.223809 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 15201
I20260618 23:24:45.224439 15827 ts_tablet_manager.cc:1905] T 5f9dae246cac497e94772de2d5a03372 P a51df27f83cb4e5cab3bcb7bf4d8efa3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:45.226001 15827 ts_tablet_manager.cc:1918] T 5f9dae246cac497e94772de2d5a03372 P a51df27f83cb4e5cab3bcb7bf4d8efa3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:24:45.226090 15827 log.cc:1196] T 5f9dae246cac497e94772de2d5a03372 P a51df27f83cb4e5cab3bcb7bf4d8efa3: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-1/wal/wals/5f9dae246cac497e94772de2d5a03372
I20260618 23:24:45.226338 15827 ts_tablet_manager.cc:1939] T 5f9dae246cac497e94772de2d5a03372 P a51df27f83cb4e5cab3bcb7bf4d8efa3: Deleting consensus metadata
I20260618 23:24:45.226548 15826 tablet_replica.cc:330] T 5f9dae246cac497e94772de2d5a03372 P e4889f468084485892fd900728fd9312: stopping tablet replica
I20260618 23:24:45.226635 15234 catalog_manager.cc:4841] TS a51df27f83cb4e5cab3bcb7bf4d8efa3 (127.8.39.130:42141): tablet 5f9dae246cac497e94772de2d5a03372 (table test-table [id=52ed044638664ec08ddd0532d753bb9b]) successfully deleted
I20260618 23:24:45.226734 15826 raft_consensus.cc:2229] T 5f9dae246cac497e94772de2d5a03372 P e4889f468084485892fd900728fd9312 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:24:45.226807 15826 raft_consensus.cc:2258] T 5f9dae246cac497e94772de2d5a03372 P e4889f468084485892fd900728fd9312 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:45.227136 15826 ts_tablet_manager.cc:1905] T 5f9dae246cac497e94772de2d5a03372 P e4889f468084485892fd900728fd9312: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:45.229516 15826 ts_tablet_manager.cc:1918] T 5f9dae246cac497e94772de2d5a03372 P e4889f468084485892fd900728fd9312: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:24:45.229616 15826 log.cc:1196] T 5f9dae246cac497e94772de2d5a03372 P e4889f468084485892fd900728fd9312: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781825053542692-8350-0/minicluster-data/ts-2/wal/wals/5f9dae246cac497e94772de2d5a03372
I20260618 23:24:45.229835 15826 ts_tablet_manager.cc:1939] T 5f9dae246cac497e94772de2d5a03372 P e4889f468084485892fd900728fd9312: Deleting consensus metadata
I20260618 23:24:45.230230 15234 catalog_manager.cc:4841] TS e4889f468084485892fd900728fd9312 (127.8.39.131:43739): tablet 5f9dae246cac497e94772de2d5a03372 (table test-table [id=52ed044638664ec08ddd0532d753bb9b]) successfully deleted
I20260618 23:24:45.232242 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 15435
I20260618 23:24:45.232340 15799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:45.232098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:45209 (local address 127.0.0.1:57506)
0618 23:24:45.232190 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:45.232306 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.129:45209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260618 23:24:45.236522 15233 catalog_manager.cc:4584] TS b46d97a84c2b4d88b79434d5e2bf1a16 (127.8.39.129:45209): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5f9dae246cac497e94772de2d5a03372: Network error: Client connection negotiation failed: client connection to 127.8.39.129:45209: connect: Connection refused (error 111)
I20260618 23:24:45.236624 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 15571
I20260618 23:24:45.241181 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 14920
I20260618 23:24:45.245973 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 15108
2026-06-18T23:24:45Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6578 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:24:45 dist-test-slave-47qt krb5kdc[15830](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:24:45 dist-test-slave-47qt krb5kdc[15830](info): set up 2 sockets
Jun 18 23:24:45 dist-test-slave-47qt krb5kdc[15830](info): commencing operation
krb5kdc: starting...
W20260618 23:24:46.321486 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:24:46 dist-test-slave-47qt krb5kdc[15830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:24:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:24:46Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.39.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.39.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.39.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.190.keytab.
Entry for principal oryx/127.8.39.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.190.keytab.
I20260618 23:24:46.444645 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:44111
--webserver_interface=127.8.39.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.190.keytab
--principal=oryx/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:39683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:44111
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:46.524777 15843 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:46.525113 15843 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:46.525171 15843 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:46.526296 15843 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:24:46.526369 15843 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260618 23:24:46.526394 15843 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:46.526423 15843 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:46.526453 15843 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:24:46.526479 15843 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:24:46.527946 15843 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:39683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:44111
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.190.keytab
--principal=oryx/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:44111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15843
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:46.528204 15843 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:46.528417 15843 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:46.533039 15848 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:46.533013 15851 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:46.533017 15849 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:46.533326 15843 server_base.cc:1034] running on GCE node
I20260618 23:24:46.533577 15843 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:46.533805 15843 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:46.534970 15843 hybrid_clock.cc:648] HybridClock initialized: now 1781825086534951 us; error 40 us; skew 500 ppm
Jun 18 23:24:46 dist-test-slave-47qt krb5kdc[15830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825086, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:46.537418 15843 init.cc:377] Logged in from keytab as oryx/127.8.39.190@KRBTEST.COM (short username oryx)
I20260618 23:24:46.537864 15843 webserver.cc:460] Webserver started at http://127.8.39.190:40337/ using document root <none> and password file <none>
I20260618 23:24:46.538086 15843 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:46.538129 15843 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:46.538271 15843 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:46.539297 15843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "71871ac5b89340e7bd6204518a23b6b8"
format_stamp: "Formatted at 2026-06-18 23:24:46 on dist-test-slave-47qt"
server_key: "62fec647aed4d3df10767f9f583880c7"
server_key_iv: "b46c721b4fce91dd5602bd5f0e4b48bb"
server_key_version: "encryptionkey@0"
I20260618 23:24:46.539645 15843 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "71871ac5b89340e7bd6204518a23b6b8"
format_stamp: "Formatted at 2026-06-18 23:24:46 on dist-test-slave-47qt"
server_key: "62fec647aed4d3df10767f9f583880c7"
server_key_iv: "b46c721b4fce91dd5602bd5f0e4b48bb"
server_key_version: "encryptionkey@0"
I20260618 23:24:46.541011 15843 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260618 23:24:46.541781 15858 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:46.541970 15843 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:46.542038 15843 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "71871ac5b89340e7bd6204518a23b6b8"
format_stamp: "Formatted at 2026-06-18 23:24:46 on dist-test-slave-47qt"
server_key: "62fec647aed4d3df10767f9f583880c7"
server_key_iv: "b46c721b4fce91dd5602bd5f0e4b48bb"
server_key_version: "encryptionkey@0"
I20260618 23:24:46.542126 15843 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:46.551402 15843 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:46.553243 15843 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:46.553398 15843 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:46.557919 15843 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:44111
I20260618 23:24:46.557965 15920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:44111 every 8 connection(s)
I20260618 23:24:46.558265 15843 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:24:46.558820 15921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:46.559329 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 15843
I20260618 23:24:46.559427 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:24:46.559645 8350 external_mini_cluster.cc:1428] Setting key 48d4ec6d84fef9f53a5c55b57212aaed
I20260618 23:24:46.561903 15921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71871ac5b89340e7bd6204518a23b6b8: Bootstrap starting.
I20260618 23:24:46.562577 15921 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71871ac5b89340e7bd6204518a23b6b8: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:46.562880 15921 log.cc:824] T 00000000000000000000000000000000 P 71871ac5b89340e7bd6204518a23b6b8: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:46.563632 15921 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71871ac5b89340e7bd6204518a23b6b8: No bootstrap required, opened a new log
I20260618 23:24:46.565663 15921 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71871ac5b89340e7bd6204518a23b6b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71871ac5b89340e7bd6204518a23b6b8" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44111 } }
I20260618 23:24:46.565794 15921 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71871ac5b89340e7bd6204518a23b6b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:46.565816 15921 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71871ac5b89340e7bd6204518a23b6b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71871ac5b89340e7bd6204518a23b6b8, State: Initialized, Role: FOLLOWER
I20260618 23:24:46.565953 15921 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71871ac5b89340e7bd6204518a23b6b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71871ac5b89340e7bd6204518a23b6b8" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44111 } }
I20260618 23:24:46.566026 15921 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71871ac5b89340e7bd6204518a23b6b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:24:46.566049 15921 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71871ac5b89340e7bd6204518a23b6b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:24:46.566114 15921 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71871ac5b89340e7bd6204518a23b6b8 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:46.566654 15921 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71871ac5b89340e7bd6204518a23b6b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71871ac5b89340e7bd6204518a23b6b8" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44111 } }
I20260618 23:24:46.566768 15921 leader_election.cc:302] T 00000000000000000000000000000000 P 71871ac5b89340e7bd6204518a23b6b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71871ac5b89340e7bd6204518a23b6b8; no voters:
I20260618 23:24:46.566951 15921 leader_election.cc:288] T 00000000000000000000000000000000 P 71871ac5b89340e7bd6204518a23b6b8 [CANDIDATE]: Term 1 election: Requested vote from peers
Jun 18 23:24:46 dist-test-slave-47qt krb5kdc[15830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM
I20260618 23:24:46.567013 15926 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71871ac5b89340e7bd6204518a23b6b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:46.567170 15926 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71871ac5b89340e7bd6204518a23b6b8 [term 1 LEADER]: Becoming Leader. State: Replica: 71871ac5b89340e7bd6204518a23b6b8, State: Running, Role: LEADER
I20260618 23:24:46.567193 15921 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71871ac5b89340e7bd6204518a23b6b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:24:46.567257 15926 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71871ac5b89340e7bd6204518a23b6b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71871ac5b89340e7bd6204518a23b6b8" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44111 } }
I20260618 23:24:46.567610 15926 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71871ac5b89340e7bd6204518a23b6b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71871ac5b89340e7bd6204518a23b6b8. Latest consensus state: current_term: 1 leader_uuid: "71871ac5b89340e7bd6204518a23b6b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71871ac5b89340e7bd6204518a23b6b8" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44111 } } }
I20260618 23:24:46.567698 15926 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71871ac5b89340e7bd6204518a23b6b8 [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:46.567799 15927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71871ac5b89340e7bd6204518a23b6b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71871ac5b89340e7bd6204518a23b6b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71871ac5b89340e7bd6204518a23b6b8" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44111 } } }
I20260618 23:24:46.568029 15927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71871ac5b89340e7bd6204518a23b6b8 [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:46.568041 15935 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:24:46.569026 15935 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:24:46.570621 15935 catalog_manager.cc:1347] Generated new cluster ID: cb4b5bdfe20144e6bf3b0ffacb5d1b13
I20260618 23:24:46.570667 15935 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:24:46.572072 15924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:46.560192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38012 (local address 127.8.39.190:44111)
0618 23:24:46.560375 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:46.560384 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:46.560425 (+ 41us) server_negotiation.cc:407] Connection header received
0618 23:24:46.560940 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:46.560949 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:46.561005 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:46.561184 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:46.561786 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:46.563129 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:46.563920 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:46.564318 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:46.564449 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:46.567571 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:46.567575 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:46.567576 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:46.567587 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:46.570500 (+ 2913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:46.570924 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:46.570927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:46.570928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:46.571025 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:46.571233 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:46.571234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:46.571235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:46.571463 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:46.571631 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:46.571731 (+ 100us) server_negotiation.cc:299] Negotiation successful
0618 23:24:46.571864 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":89,"threads_started":1}
I20260618 23:24:46.591777 15935 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:24:46.592885 15935 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:24:46.597995 15935 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71871ac5b89340e7bd6204518a23b6b8: Generated new TSK 0
I20260618 23:24:46.598187 15935 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.39.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.39.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.39.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.129.keytab.
Entry for principal oryx/127.8.39.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.129.keytab.
I20260618 23:24:46.627787 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.129:0
--local_ip_for_outbound_sockets=127.8.39.129
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:44111
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.129.keytab
--principal=oryx/127.8.39.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:39683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:46.716791 15947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:46.716988 15947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:46.717015 15947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:46.718245 15947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:46.718287 15947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:46.718308 15947 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 23:24:46.718379 15947 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.129
I20260618 23:24:46.719934 15947 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:39683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.129.keytab
--principal=oryx/127.8.39.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.39.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.39.190:44111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15947
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.129
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:46.720183 15947 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:46.720618 15947 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:46.725746 15955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:46.725747 15952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:46.725804 15947 server_base.cc:1034] running on GCE node
W20260618 23:24:46.725746 15953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:46.726200 15947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:46.726465 15947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:46.727600 15947 hybrid_clock.cc:648] HybridClock initialized: now 1781825086727576 us; error 39 us; skew 500 ppm
Jun 18 23:24:46 dist-test-slave-47qt krb5kdc[15830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825086, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:46.730496 15947 init.cc:377] Logged in from keytab as oryx/127.8.39.129@KRBTEST.COM (short username oryx)
I20260618 23:24:46.730933 15947 webserver.cc:460] Webserver started at http://127.8.39.129:41261/ using document root <none> and password file <none>
I20260618 23:24:46.731163 15947 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:46.731230 15947 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:46.731379 15947 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:46.732373 15947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-0/data/instance:
uuid: "3cffed2d98bf4816b9d3d4a263e62b95"
format_stamp: "Formatted at 2026-06-18 23:24:46 on dist-test-slave-47qt"
server_key: "1cc3061c389d68ef3930a5ecd4c68684"
server_key_iv: "77422e4c9b305437e766d71225585b92"
server_key_version: "encryptionkey@0"
I20260618 23:24:46.732720 15947 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance:
uuid: "3cffed2d98bf4816b9d3d4a263e62b95"
format_stamp: "Formatted at 2026-06-18 23:24:46 on dist-test-slave-47qt"
server_key: "1cc3061c389d68ef3930a5ecd4c68684"
server_key_iv: "77422e4c9b305437e766d71225585b92"
server_key_version: "encryptionkey@0"
I20260618 23:24:46.734109 15947 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:24:46.734813 15962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:46.734987 15947 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:46.735083 15947 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-0/wal
uuid: "3cffed2d98bf4816b9d3d4a263e62b95"
format_stamp: "Formatted at 2026-06-18 23:24:46 on dist-test-slave-47qt"
server_key: "1cc3061c389d68ef3930a5ecd4c68684"
server_key_iv: "77422e4c9b305437e766d71225585b92"
server_key_version: "encryptionkey@0"
I20260618 23:24:46.735162 15947 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:46.752699 15947 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:46.754858 15947 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:46.755041 15947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:46.755568 15947 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:46.755630 15947 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:46.755663 15947 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:46.755716 15947 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:46.762501 15947 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.129:40645
I20260618 23:24:46.762566 16077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.129:40645 every 8 connection(s)
I20260618 23:24:46.762887 15947 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
Jun 18 23:24:46 dist-test-slave-47qt krb5kdc[15830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825086, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM
I20260618 23:24:46.763183 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 15947
I20260618 23:24:46.763298 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance
I20260618 23:24:46.763532 8350 external_mini_cluster.cc:1428] Setting key 36e92c3612b742c5131a8fc6feecacae
I20260618 23:24:46.766575 15924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:46.756197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:59919 (local address 127.8.39.190:44111)
0618 23:24:46.756261 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:46.756263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:46.756468 (+ 205us) server_negotiation.cc:407] Connection header received
0618 23:24:46.757019 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:46.757021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:46.757034 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:46.757109 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:46.758032 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:46.758874 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:46.760440 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:46.760767 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:46.760853 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:46.763499 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:46.763500 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:46.763501 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:46.763509 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:46.764986 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:46.765523 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:46.765524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:46.765525 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:46.765595 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:46.765884 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:46.765884 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:46.765885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:46.766033 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:46.766103 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:46.766381 (+ 278us) server_negotiation.cc:299] Negotiation successful
0618 23:24:46.766477 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260618 23:24:46.766834 15971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:46.756194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:44111 (local address 127.8.39.129:59919)
0618 23:24:46.756334 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:46.756366 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:46.756865 (+ 499us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:46.757260 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:46.757264 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:46.757647 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:46.757931 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:46.757936 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:46.758981 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:46.758982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:46.760324 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:46.760327 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:46.760869 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:46.760870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:46.760977 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:46.761592 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:46.761603 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:46.763386 (+ 1783us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:46.765069 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:46.765075 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:46.765078 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:46.765416 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:46.765692 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:46.765693 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:46.765693 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:46.765798 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:46.766117 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:46.766123 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:46.766231 (+ 108us) client_negotiation.cc:769] Sending connection context
0618 23:24:46.766320 (+ 89us) client_negotiation.cc:240] Negotiation successful
0618 23:24:46.766486 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":74,"threads_started":1}
I20260618 23:24:46.767539 16078 heartbeater.cc:347] Connected to a master server at 127.8.39.190:44111
I20260618 23:24:46.767758 16078 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:46.767946 16078 heartbeater.cc:511] Master 127.8.39.190:44111 requested a full tablet report, sending...
I20260618 23:24:46.768720 15874 ts_manager.cc:194] Registered new tserver with Master: 3cffed2d98bf4816b9d3d4a263e62b95 (127.8.39.129:40645)
I20260618 23:24:46.769876 15874 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.39.129@KRBTEST.COM'} at 127.8.39.129:59919
WARNING: no policy specified for oryx/127.8.39.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.39.130@KRBTEST.COM" created.
I20260618 23:24:46.775914 15924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:46.769175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:35703 (local address 127.8.39.190:44111)
0618 23:24:46.769306 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:46.769308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:46.770232 (+ 924us) server_negotiation.cc:407] Connection header received
0618 23:24:46.770304 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:46.770305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:46.770317 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:46.770384 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:46.770811 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:46.771321 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:46.772417 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:46.772591 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:46.772652 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:46.773553 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:46.773555 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:46.773555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:46.773561 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:46.774818 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:46.775116 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:46.775117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:46.775118 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:46.775176 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:46.775395 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:46.775396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:46.775397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:46.775522 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:46.775574 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:46.775761 (+ 187us) server_negotiation.cc:299] Negotiation successful
0618 23:24:46.775828 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260618 23:24:46.775933 16085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:46.769063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:44111 (local address 127.8.39.129:35703)
0618 23:24:46.770157 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:46.770166 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:46.770246 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:46.770391 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:46.770393 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:46.770650 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:46.770719 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:46.770721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:46.771424 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:46.771425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:46.772320 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:46.772323 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:46.772675 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:46.772677 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:46.772756 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:46.773207 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:46.773211 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:46.773471 (+ 260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:46.774887 (+ 1416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:46.774888 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:46.774888 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:46.775045 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:46.775249 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:46.775250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:46.775250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:46.775311 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:46.775581 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:46.775582 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:46.775629 (+ 47us) client_negotiation.cc:769] Sending connection context
0618 23:24:46.775728 (+ 99us) client_negotiation.cc:240] Negotiation successful
0618 23:24:46.775835 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1071,"thread_start_us":104,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.39.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.130.keytab.
Entry for principal oryx/127.8.39.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.130.keytab.
I20260618 23:24:46.791464 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.130:0
--local_ip_for_outbound_sockets=127.8.39.130
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:44111
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.130.keytab
--principal=oryx/127.8.39.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:39683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:46.873510 16087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:46.873749 16087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:46.873790 16087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:46.874939 16087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:46.875005 16087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:46.875043 16087 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 23:24:46.875095 16087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.130
I20260618 23:24:46.876559 16087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:39683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.130.keytab
--principal=oryx/127.8.39.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.39.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.39.190:44111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.130
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:46.876814 16087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:46.877044 16087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:46.881997 16093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:46.882018 16087 server_base.cc:1034] running on GCE node
W20260618 23:24:46.881991 16092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:46.881991 16095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:46.882534 16087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:46.882769 16087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:46.883951 16087 hybrid_clock.cc:648] HybridClock initialized: now 1781825086883939 us; error 43 us; skew 500 ppm
Jun 18 23:24:46 dist-test-slave-47qt krb5kdc[15830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825086, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:46.886658 16087 init.cc:377] Logged in from keytab as oryx/127.8.39.130@KRBTEST.COM (short username oryx)
I20260618 23:24:46.887081 16087 webserver.cc:460] Webserver started at http://127.8.39.130:46109/ using document root <none> and password file <none>
I20260618 23:24:46.887298 16087 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:46.887374 16087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:46.887624 16087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:46.888634 16087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-1/data/instance:
uuid: "b725d44cdbe148c2864a6b795ab02019"
format_stamp: "Formatted at 2026-06-18 23:24:46 on dist-test-slave-47qt"
server_key: "50146c90b419f11d163b0c2f0aca2ad1"
server_key_iv: "fcb9e5b1da0ffe7c1c269613d98b158f"
server_key_version: "encryptionkey@0"
I20260618 23:24:46.888988 16087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance:
uuid: "b725d44cdbe148c2864a6b795ab02019"
format_stamp: "Formatted at 2026-06-18 23:24:46 on dist-test-slave-47qt"
server_key: "50146c90b419f11d163b0c2f0aca2ad1"
server_key_iv: "fcb9e5b1da0ffe7c1c269613d98b158f"
server_key_version: "encryptionkey@0"
I20260618 23:24:46.890298 16087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:24:46.891088 16102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:46.891264 16087 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:24:46.891383 16087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-1/wal
uuid: "b725d44cdbe148c2864a6b795ab02019"
format_stamp: "Formatted at 2026-06-18 23:24:46 on dist-test-slave-47qt"
server_key: "50146c90b419f11d163b0c2f0aca2ad1"
server_key_iv: "fcb9e5b1da0ffe7c1c269613d98b158f"
server_key_version: "encryptionkey@0"
I20260618 23:24:46.891455 16087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:46.902009 16087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:46.903944 16087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:46.904100 16087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:46.904556 16087 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:46.904618 16087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:46.904659 16087 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:46.904701 16087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:46.911993 16087 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.130:38975
I20260618 23:24:46.912091 16217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.130:38975 every 8 connection(s)
I20260618 23:24:46.912382 16087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
Jun 18 23:24:46 dist-test-slave-47qt krb5kdc[15830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825086, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM
I20260618 23:24:46.915330 15924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:46.905062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:52249 (local address 127.8.39.190:44111)
0618 23:24:46.905171 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:46.905173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:46.905350 (+ 177us) server_negotiation.cc:407] Connection header received
0618 23:24:46.906057 (+ 707us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:46.906058 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:46.906069 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:46.906130 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:46.907027 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:46.907590 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:46.909234 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:46.909451 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:46.909538 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:46.913034 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:46.913035 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:46.913035 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:46.913041 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:46.914254 (+ 1213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:46.914617 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:46.914618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:46.914618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:46.914664 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:46.914866 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:46.914867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:46.914867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:46.914972 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:46.915022 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:46.915218 (+ 196us) server_negotiation.cc:299] Negotiation successful
0618 23:24:46.915271 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260618 23:24:46.915580 16111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:46.905030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:44111 (local address 127.8.39.130:52249)
0618 23:24:46.905205 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:46.905253 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:46.905953 (+ 700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:46.906207 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:46.906212 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:46.906618 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:46.906923 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:46.906929 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:46.907690 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:46.907691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:46.909153 (+ 1462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:46.909157 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:46.909557 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:46.909558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:46.909684 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:46.911072 (+ 1388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:46.911158 (+ 86us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:46.912915 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:46.914326 (+ 1411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:46.914332 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:46.914334 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:46.914549 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:46.914722 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:46.914723 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:46.914723 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:46.914813 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:46.915026 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:46.915030 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:46.915131 (+ 101us) client_negotiation.cc:769] Sending connection context
0618 23:24:46.915209 (+ 78us) client_negotiation.cc:240] Negotiation successful
0618 23:24:46.915321 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":112,"threads_started":1}
I20260618 23:24:46.916088 16218 heartbeater.cc:347] Connected to a master server at 127.8.39.190:44111
I20260618 23:24:46.916188 16218 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:46.916272 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 16087
I20260618 23:24:46.916383 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance
I20260618 23:24:46.916379 16218 heartbeater.cc:511] Master 127.8.39.190:44111 requested a full tablet report, sending...
I20260618 23:24:46.916615 8350 external_mini_cluster.cc:1428] Setting key 7a3e46ba9e33db373c11260520e000fb
I20260618 23:24:46.916751 15875 ts_manager.cc:194] Registered new tserver with Master: b725d44cdbe148c2864a6b795ab02019 (127.8.39.130:38975)
I20260618 23:24:46.917279 15875 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.39.130@KRBTEST.COM'} at 127.8.39.130:52249
I20260618 23:24:46.924021 15924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:46.917217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:49553 (local address 127.8.39.190:44111)
0618 23:24:46.917306 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:46.917308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:46.917422 (+ 114us) server_negotiation.cc:407] Connection header received
0618 23:24:46.917520 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:46.917521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:46.917531 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:46.917588 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:46.917933 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:46.918669 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:46.919713 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:46.919897 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:46.919958 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:46.920982 (+ 1024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:46.920984 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:46.920984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:46.920990 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:46.922761 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:46.923141 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:46.923142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:46.923142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:46.923262 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:46.923526 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:46.923527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:46.923527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:46.923654 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:46.923707 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:46.923903 (+ 196us) server_negotiation.cc:299] Negotiation successful
0618 23:24:46.923965 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260618 23:24:46.924131 16224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:46.917164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:44111 (local address 127.8.39.130:49553)
0618 23:24:46.917365 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:46.917376 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:46.917446 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:46.917607 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:46.917608 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:46.917792 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:46.917856 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:46.917858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:46.918773 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:46.918774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:46.919619 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:46.919622 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:46.919978 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:46.919979 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:46.920054 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:46.920487 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:46.920492 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:46.920884 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:46.922853 (+ 1969us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:46.922854 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:46.922854 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:46.923068 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:46.923365 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:46.923366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:46.923367 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:46.923447 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:46.923730 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:46.923731 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:46.923790 (+ 59us) client_negotiation.cc:769] Sending connection context
0618 23:24:46.923878 (+ 88us) client_negotiation.cc:240] Negotiation successful
0618 23:24:46.923999 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":63,"threads_started":1}
WARNING: no policy specified for oryx/127.8.39.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.39.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.39.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.131.keytab.
Entry for principal oryx/127.8.39.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.131.keytab.
I20260618 23:24:46.944386 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.131:0
--local_ip_for_outbound_sockets=127.8.39.131
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:44111
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.131.keytab
--principal=oryx/127.8.39.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:39683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:47.024514 16227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:47.024705 16227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:47.024729 16227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:47.025938 16227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:47.025977 16227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:47.025998 16227 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260618 23:24:47.026057 16227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.131
I20260618 23:24:47.027493 16227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:39683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.131.keytab
--principal=oryx/127.8.39.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.39.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.39.190:44111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.131
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:47.027736 16227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:47.027959 16227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:47.033015 16233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:47.033011 16235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:47.033196 16227 server_base.cc:1034] running on GCE node
W20260618 23:24:47.033015 16232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:47.033560 16227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:47.033805 16227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:47.035007 16227 hybrid_clock.cc:648] HybridClock initialized: now 1781825087034999 us; error 37 us; skew 500 ppm
Jun 18 23:24:47 dist-test-slave-47qt krb5kdc[15830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825087, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:47.037703 16227 init.cc:377] Logged in from keytab as oryx/127.8.39.131@KRBTEST.COM (short username oryx)
I20260618 23:24:47.038189 16227 webserver.cc:460] Webserver started at http://127.8.39.131:36941/ using document root <none> and password file <none>
I20260618 23:24:47.038488 16227 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:47.038535 16227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:47.038638 16227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:47.039826 16227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-2/data/instance:
uuid: "adc409b2a6554449a338c9c17ca51855"
format_stamp: "Formatted at 2026-06-18 23:24:47 on dist-test-slave-47qt"
server_key: "c395474e64d6c87ae5351ee14eb93ce2"
server_key_iv: "110b2b6b45906321e63601f024cd4e81"
server_key_version: "encryptionkey@0"
I20260618 23:24:47.040259 16227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance:
uuid: "adc409b2a6554449a338c9c17ca51855"
format_stamp: "Formatted at 2026-06-18 23:24:47 on dist-test-slave-47qt"
server_key: "c395474e64d6c87ae5351ee14eb93ce2"
server_key_iv: "110b2b6b45906321e63601f024cd4e81"
server_key_version: "encryptionkey@0"
I20260618 23:24:47.041671 16227 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260618 23:24:47.042690 16242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:47.043133 16227 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:47.043221 16227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-2/wal
uuid: "adc409b2a6554449a338c9c17ca51855"
format_stamp: "Formatted at 2026-06-18 23:24:47 on dist-test-slave-47qt"
server_key: "c395474e64d6c87ae5351ee14eb93ce2"
server_key_iv: "110b2b6b45906321e63601f024cd4e81"
server_key_version: "encryptionkey@0"
I20260618 23:24:47.043318 16227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:47.058290 16227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:47.060190 16227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:47.060297 16227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:47.060820 16227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:47.060858 16227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:47.060881 16227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:47.060943 16227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 18 23:24:47 dist-test-slave-47qt krb5kdc[15830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825087, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM
I20260618 23:24:47.067965 16227 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.131:43301
I20260618 23:24:47.068228 16357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.131:43301 every 8 connection(s)
I20260618 23:24:47.068372 16227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
I20260618 23:24:47.069217 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 16227
I20260618 23:24:47.069347 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance
I20260618 23:24:47.069573 8350 external_mini_cluster.cc:1428] Setting key e9bf6d644efce250cf1f34cb649316c8
I20260618 23:24:47.070993 15924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:47.061429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:39877 (local address 127.8.39.190:44111)
0618 23:24:47.061479 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:47.061483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:47.061721 (+ 238us) server_negotiation.cc:407] Connection header received
0618 23:24:47.062262 (+ 541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:47.062264 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:47.062275 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:47.062330 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:47.062971 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.063549 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.064598 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.064937 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.065004 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:47.067781 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:47.067782 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:47.067782 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:47.067790 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:47.069666 (+ 1876us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:47.070112 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:47.070112 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:47.070113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:47.070177 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:47.070435 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:47.070436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:47.070436 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:47.070577 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:47.070639 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:47.070871 (+ 232us) server_negotiation.cc:299] Negotiation successful
0618 23:24:47.070933 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260618 23:24:47.071135 16251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:47.061444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:44111 (local address 127.8.39.131:39877)
0618 23:24:47.061602 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:47.061634 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:47.062173 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:47.062402 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:47.062406 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:47.062687 (+ 281us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:47.062864 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:47.062868 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.063639 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.063640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:47.064498 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:47.064501 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.065009 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.065010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:47.065165 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:47.065688 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:47.065697 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:47.067681 (+ 1984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:47.069741 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:47.069748 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:47.069750 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:47.070040 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:47.070249 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:47.070250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:47.070250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:47.070363 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:47.070650 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:47.070655 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:47.070763 (+ 108us) client_negotiation.cc:769] Sending connection context
0618 23:24:47.070843 (+ 80us) client_negotiation.cc:240] Negotiation successful
0618 23:24:47.070955 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":76,"threads_started":1}
I20260618 23:24:47.071559 16358 heartbeater.cc:347] Connected to a master server at 127.8.39.190:44111
I20260618 23:24:47.071637 16358 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:47.071817 16358 heartbeater.cc:511] Master 127.8.39.190:44111 requested a full tablet report, sending...
I20260618 23:24:47.072204 15874 ts_manager.cc:194] Registered new tserver with Master: adc409b2a6554449a338c9c17ca51855 (127.8.39.131:43301)
I20260618 23:24:47.072679 15874 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.39.131@KRBTEST.COM'} at 127.8.39.131:39877
I20260618 23:24:47.077669 15924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:47.072633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:36035 (local address 127.8.39.190:44111)
0618 23:24:47.072666 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:47.072667 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:47.072806 (+ 139us) server_negotiation.cc:407] Connection header received
0618 23:24:47.072827 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:47.072828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:47.072838 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:47.072926 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:47.073308 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.073742 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.074389 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.074562 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.074615 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:47.075443 (+ 828us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:47.075445 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:47.075445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:47.075450 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:47.076625 (+ 1175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:47.076958 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:47.076959 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:47.076960 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:47.077003 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:47.077225 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:47.077226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:47.077226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:47.077311 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:47.077359 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:47.077561 (+ 202us) server_negotiation.cc:299] Negotiation successful
0618 23:24:47.077615 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260618 23:24:47.077675 16364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:47.072583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:44111 (local address 127.8.39.131:36035)
0618 23:24:47.072710 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:47.072720 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:47.072777 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:47.072934 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:47.072935 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:47.073188 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:47.073243 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:47.073245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.073825 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.073825 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:47.074318 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:47.074319 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.074654 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.074654 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:47.074714 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:47.075112 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:47.075116 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:47.075363 (+ 247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:47.076711 (+ 1348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:47.076712 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:47.076712 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:47.076890 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:47.077062 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:47.077062 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:47.077063 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:47.077113 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:47.077420 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:47.077421 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:47.077461 (+ 40us) client_negotiation.cc:769] Sending connection context
0618 23:24:47.077528 (+ 67us) client_negotiation.cc:240] Negotiation successful
0618 23:24:47.077606 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":62,"threads_started":1}
I20260618 23:24:47.081313 8350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 18 23:24:47 dist-test-slave-47qt krb5kdc[15830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825087, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 18 23:24:47 dist-test-slave-47qt krb5kdc[15830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:47 dist-test-slave-47qt krb5kdc[15830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
W20260618 23:24:47.102766 15924 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:47.096678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38016 (local address 127.8.39.190:44111)
0618 23:24:47.096741 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:47.096743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:47.096751 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:24:47.096772 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:47.096772 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:47.096786 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:47.096834 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:47.097428 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.097910 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.098780 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.098969 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.099032 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:47.102654 (+ 3622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38016: BlockingRecv error: recv got EOF from 127.0.0.1:38016 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 18 23:24:47 dist-test-slave-47qt krb5kdc[15830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825087, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM
I20260618 23:24:47.110970 15924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:47.103593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38028 (local address 127.8.39.190:44111)
0618 23:24:47.103692 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:47.103694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:47.103772 (+ 78us) server_negotiation.cc:407] Connection header received
0618 23:24:47.103849 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:47.103849 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:47.103862 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:47.103943 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:47.104559 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.105013 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.105707 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.105891 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.105996 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:47.108482 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:47.108483 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:47.108484 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:47.108491 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:47.109875 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:47.110231 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:47.110232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:47.110233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:47.110292 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:47.110493 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:47.110494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:47.110494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:47.110598 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:47.110702 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:47.110805 (+ 103us) server_negotiation.cc:299] Negotiation successful
0618 23:24:47.110873 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260618 23:24:47.112516 15874 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:24:47.114161 15874 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:24:47.120626 16382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:47.116435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:40645 (local address 127.0.0.1:41860)
0618 23:24:47.116868 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:47.116877 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:47.116925 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:47.117172 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:47.117173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:47.117175 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:47.117508 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:47.117510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.118665 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.118666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:47.119378 (+ 712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:47.119379 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.120193 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.120194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:47.120283 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:47.120297 (+ 14us) client_negotiation.cc:769] Sending connection context
0618 23:24:47.120480 (+ 183us) client_negotiation.cc:240] Negotiation successful
0618 23:24:47.120544 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"mutex_wait_us":2,"thread_start_us":75,"threads_started":1}
I20260618 23:24:47.120765 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:47.116637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47048 (local address 127.8.39.131:43301)
0618 23:24:47.116794 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:47.116799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:47.116975 (+ 176us) server_negotiation.cc:407] Connection header received
0618 23:24:47.117018 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:47.117020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:47.117036 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:47.117106 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:47.117349 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.118323 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.119739 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.120392 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.120472 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:47.120541 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:47.120635 (+ 94us) server_negotiation.cc:299] Negotiation successful
0618 23:24:47.120666 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":68,"threads_started":1}
I20260618 23:24:47.120921 16380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:47.116376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:38975 (local address 127.0.0.1:53670)
0618 23:24:47.116612 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:47.116647 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:47.116741 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:47.116932 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:47.116935 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:47.116937 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:47.116997 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:47.116999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.118078 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.118079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:47.119742 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:47.119744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.120693 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.120694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:47.120763 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:47.120767 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:24:47.120816 (+ 49us) client_negotiation.cc:240] Negotiation successful
0618 23:24:47.120859 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":68,"threads_started":1}
I20260618 23:24:47.121223 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:47.116454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53670 (local address 127.8.39.130:38975)
0618 23:24:47.116590 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:47.116595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:47.116722 (+ 127us) server_negotiation.cc:407] Connection header received
0618 23:24:47.116823 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:47.116825 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:47.116840 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:47.116930 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:47.117072 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.117981 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.119809 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.120531 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.120592 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:47.120658 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:47.121030 (+ 372us) server_negotiation.cc:299] Negotiation successful
0618 23:24:47.121073 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
I20260618 23:24:47.121266 16385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:47.116638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41860 (local address 127.8.39.129:40645)
0618 23:24:47.116794 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:47.116799 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:47.116982 (+ 183us) server_negotiation.cc:407] Connection header received
0618 23:24:47.117018 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:47.117020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:47.117035 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:47.117174 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:47.117586 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.118577 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.119467 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.120103 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.120177 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:47.120247 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:47.121110 (+ 863us) server_negotiation.cc:299] Negotiation successful
0618 23:24:47.121178 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":68,"threads_started":1}
I20260618 23:24:47.121414 16381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:47.116435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:43301 (local address 127.0.0.1:47048)
0618 23:24:47.116844 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:47.116854 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:47.116913 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:47.117156 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:47.117157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:47.117159 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:47.117215 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:47.117216 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.118430 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.118431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:47.119646 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:47.119649 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.120487 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.120488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:47.120555 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:47.120559 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:24:47.120612 (+ 53us) client_negotiation.cc:240] Negotiation successful
0618 23:24:47.120653 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":74,"threads_started":1}
I20260618 23:24:47.121891 16012 tablet_service.cc:1467] Processing CreateTablet for tablet 3d19cf757a874fe28fa14f60b7b48e4c (DEFAULT_TABLE table=test-table [id=1e6824e7d6134a19b73b69aa2ae23d35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:47.121892 16152 tablet_service.cc:1467] Processing CreateTablet for tablet 3d19cf757a874fe28fa14f60b7b48e4c (DEFAULT_TABLE table=test-table [id=1e6824e7d6134a19b73b69aa2ae23d35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:47.121946 16278 tablet_service.cc:1467] Processing CreateTablet for tablet 3d19cf757a874fe28fa14f60b7b48e4c (DEFAULT_TABLE table=test-table [id=1e6824e7d6134a19b73b69aa2ae23d35]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:47.122257 16278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d19cf757a874fe28fa14f60b7b48e4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:47.122257 16152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d19cf757a874fe28fa14f60b7b48e4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:47.122268 16012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d19cf757a874fe28fa14f60b7b48e4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:47.125319 16386 tablet_bootstrap.cc:492] T 3d19cf757a874fe28fa14f60b7b48e4c P 3cffed2d98bf4816b9d3d4a263e62b95: Bootstrap starting.
I20260618 23:24:47.125753 16387 tablet_bootstrap.cc:492] T 3d19cf757a874fe28fa14f60b7b48e4c P adc409b2a6554449a338c9c17ca51855: Bootstrap starting.
I20260618 23:24:47.126103 16386 tablet_bootstrap.cc:654] T 3d19cf757a874fe28fa14f60b7b48e4c P 3cffed2d98bf4816b9d3d4a263e62b95: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:47.126319 16387 tablet_bootstrap.cc:654] T 3d19cf757a874fe28fa14f60b7b48e4c P adc409b2a6554449a338c9c17ca51855: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:47.126443 16386 log.cc:824] T 3d19cf757a874fe28fa14f60b7b48e4c P 3cffed2d98bf4816b9d3d4a263e62b95: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:47.126645 16387 log.cc:824] T 3d19cf757a874fe28fa14f60b7b48e4c P adc409b2a6554449a338c9c17ca51855: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:47.126791 16388 tablet_bootstrap.cc:492] T 3d19cf757a874fe28fa14f60b7b48e4c P b725d44cdbe148c2864a6b795ab02019: Bootstrap starting.
I20260618 23:24:47.127288 16386 tablet_bootstrap.cc:492] T 3d19cf757a874fe28fa14f60b7b48e4c P 3cffed2d98bf4816b9d3d4a263e62b95: No bootstrap required, opened a new log
I20260618 23:24:47.127368 16388 tablet_bootstrap.cc:654] T 3d19cf757a874fe28fa14f60b7b48e4c P b725d44cdbe148c2864a6b795ab02019: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:47.127368 16386 ts_tablet_manager.cc:1397] T 3d19cf757a874fe28fa14f60b7b48e4c P 3cffed2d98bf4816b9d3d4a263e62b95: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:24:47.127511 16387 tablet_bootstrap.cc:492] T 3d19cf757a874fe28fa14f60b7b48e4c P adc409b2a6554449a338c9c17ca51855: No bootstrap required, opened a new log
I20260618 23:24:47.127660 16388 log.cc:824] T 3d19cf757a874fe28fa14f60b7b48e4c P b725d44cdbe148c2864a6b795ab02019: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:47.127688 16387 ts_tablet_manager.cc:1397] T 3d19cf757a874fe28fa14f60b7b48e4c P adc409b2a6554449a338c9c17ca51855: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:24:47.128584 16388 tablet_bootstrap.cc:492] T 3d19cf757a874fe28fa14f60b7b48e4c P b725d44cdbe148c2864a6b795ab02019: No bootstrap required, opened a new log
I20260618 23:24:47.128651 16388 ts_tablet_manager.cc:1397] T 3d19cf757a874fe28fa14f60b7b48e4c P b725d44cdbe148c2864a6b795ab02019: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260618 23:24:47.129240 16386 raft_consensus.cc:348] T 3d19cf757a874fe28fa14f60b7b48e4c P 3cffed2d98bf4816b9d3d4a263e62b95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:47.129411 16386 raft_consensus.cc:374] T 3d19cf757a874fe28fa14f60b7b48e4c P 3cffed2d98bf4816b9d3d4a263e62b95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:47.129453 16386 raft_consensus.cc:729] T 3d19cf757a874fe28fa14f60b7b48e4c P 3cffed2d98bf4816b9d3d4a263e62b95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cffed2d98bf4816b9d3d4a263e62b95, State: Initialized, Role: FOLLOWER
I20260618 23:24:47.129560 16386 consensus_queue.cc:260] T 3d19cf757a874fe28fa14f60b7b48e4c P 3cffed2d98bf4816b9d3d4a263e62b95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:47.129848 16078 heartbeater.cc:503] Master 127.8.39.190:44111 was elected leader, sending a full tablet report...
I20260618 23:24:47.129848 16386 ts_tablet_manager.cc:1428] T 3d19cf757a874fe28fa14f60b7b48e4c P 3cffed2d98bf4816b9d3d4a263e62b95: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260618 23:24:47.129869 16387 raft_consensus.cc:348] T 3d19cf757a874fe28fa14f60b7b48e4c P adc409b2a6554449a338c9c17ca51855 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:47.130050 16387 raft_consensus.cc:374] T 3d19cf757a874fe28fa14f60b7b48e4c P adc409b2a6554449a338c9c17ca51855 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:47.130096 16387 raft_consensus.cc:729] T 3d19cf757a874fe28fa14f60b7b48e4c P adc409b2a6554449a338c9c17ca51855 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adc409b2a6554449a338c9c17ca51855, State: Initialized, Role: FOLLOWER
I20260618 23:24:47.130247 16387 consensus_queue.cc:260] T 3d19cf757a874fe28fa14f60b7b48e4c P adc409b2a6554449a338c9c17ca51855 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:47.130522 16387 ts_tablet_manager.cc:1428] T 3d19cf757a874fe28fa14f60b7b48e4c P adc409b2a6554449a338c9c17ca51855: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:24:47.130553 16358 heartbeater.cc:503] Master 127.8.39.190:44111 was elected leader, sending a full tablet report...
I20260618 23:24:47.130868 16388 raft_consensus.cc:348] T 3d19cf757a874fe28fa14f60b7b48e4c P b725d44cdbe148c2864a6b795ab02019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:47.131036 16388 raft_consensus.cc:374] T 3d19cf757a874fe28fa14f60b7b48e4c P b725d44cdbe148c2864a6b795ab02019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:47.131083 16388 raft_consensus.cc:729] T 3d19cf757a874fe28fa14f60b7b48e4c P b725d44cdbe148c2864a6b795ab02019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b725d44cdbe148c2864a6b795ab02019, State: Initialized, Role: FOLLOWER
I20260618 23:24:47.131192 16388 consensus_queue.cc:260] T 3d19cf757a874fe28fa14f60b7b48e4c P b725d44cdbe148c2864a6b795ab02019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:47.131467 16388 ts_tablet_manager.cc:1428] T 3d19cf757a874fe28fa14f60b7b48e4c P b725d44cdbe148c2864a6b795ab02019: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:24:47.131518 16218 heartbeater.cc:503] Master 127.8.39.190:44111 was elected leader, sending a full tablet report...
W20260618 23:24:47.163010 16219 tablet.cc:2369] T 3d19cf757a874fe28fa14f60b7b48e4c P b725d44cdbe148c2864a6b795ab02019: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:24:47.170400 16392 raft_consensus.cc:482] T 3d19cf757a874fe28fa14f60b7b48e4c P 3cffed2d98bf4816b9d3d4a263e62b95 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:24:47.170527 16392 raft_consensus.cc:504] T 3d19cf757a874fe28fa14f60b7b48e4c P 3cffed2d98bf4816b9d3d4a263e62b95 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:47.170881 16392 leader_election.cc:288] T 3d19cf757a874fe28fa14f60b7b48e4c P 3cffed2d98bf4816b9d3d4a263e62b95 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b725d44cdbe148c2864a6b795ab02019 (127.8.39.130:38975), adc409b2a6554449a338c9c17ca51855 (127.8.39.131:43301)
I20260618 23:24:47.173830 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:47.171263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:57939 (local address 127.8.39.131:43301)
0618 23:24:47.171304 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:47.171306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:47.171317 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:24:47.171347 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:47.171348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:47.171358 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:47.171427 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:47.171584 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.172048 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.173078 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.173471 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.173563 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:47.173573 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:47.173727 (+ 154us) server_negotiation.cc:299] Negotiation successful
0618 23:24:47.173762 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260618 23:24:47.173848 15971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:47.171086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:43301 (local address 127.8.39.129:57939)
0618 23:24:47.171217 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:47.171246 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:47.171292 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:47.171443 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:47.171444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:47.171445 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:47.171517 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:47.171519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.172145 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.172146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:47.172990 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:47.172992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.173563 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.173564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:47.173641 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:47.173644 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:24:47.173719 (+ 75us) client_negotiation.cc:240] Negotiation successful
0618 23:24:47.173763 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":50}
I20260618 23:24:47.173969 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:47.171144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:53719 (local address 127.8.39.130:38975)
0618 23:24:47.171213 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:47.171215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:47.171393 (+ 178us) server_negotiation.cc:407] Connection header received
0618 23:24:47.171436 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:47.171437 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:47.171449 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:47.171516 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:47.171670 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.172166 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.173231 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.173587 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.173694 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:47.173701 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:47.173856 (+ 155us) server_negotiation.cc:299] Negotiation successful
0618 23:24:47.173890 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260618 23:24:47.174155 16312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d19cf757a874fe28fa14f60b7b48e4c" candidate_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc409b2a6554449a338c9c17ca51855" is_pre_election: true
I20260618 23:24:47.174306 16312 raft_consensus.cc:2454] T 3d19cf757a874fe28fa14f60b7b48e4c P adc409b2a6554449a338c9c17ca51855 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cffed2d98bf4816b9d3d4a263e62b95 in term 0.
I20260618 23:24:47.174377 16395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:47.171113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:38975 (local address 127.8.39.129:53719)
0618 23:24:47.171284 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:47.171305 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:47.171376 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:47.171519 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:47.171520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:47.171521 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:47.171616 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:47.171617 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.172242 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.172243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:47.173107 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:47.173110 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.173692 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.173693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:47.173762 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:47.173766 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:24:47.174207 (+ 441us) client_negotiation.cc:240] Negotiation successful
0618 23:24:47.174242 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":123,"threads_started":1}
I20260618 23:24:47.174556 15963 leader_election.cc:302] T 3d19cf757a874fe28fa14f60b7b48e4c P 3cffed2d98bf4816b9d3d4a263e62b95 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cffed2d98bf4816b9d3d4a263e62b95, adc409b2a6554449a338c9c17ca51855; no voters:
I20260618 23:24:47.174681 16392 raft_consensus.cc:2781] T 3d19cf757a874fe28fa14f60b7b48e4c P 3cffed2d98bf4816b9d3d4a263e62b95 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:24:47.174736 16392 raft_consensus.cc:482] T 3d19cf757a874fe28fa14f60b7b48e4c P 3cffed2d98bf4816b9d3d4a263e62b95 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:24:47.174762 16392 raft_consensus.cc:3037] T 3d19cf757a874fe28fa14f60b7b48e4c P 3cffed2d98bf4816b9d3d4a263e62b95 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:47.174777 16172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d19cf757a874fe28fa14f60b7b48e4c" candidate_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b725d44cdbe148c2864a6b795ab02019" is_pre_election: true
I20260618 23:24:47.174925 16172 raft_consensus.cc:2454] T 3d19cf757a874fe28fa14f60b7b48e4c P b725d44cdbe148c2864a6b795ab02019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cffed2d98bf4816b9d3d4a263e62b95 in term 0.
I20260618 23:24:47.175781 16392 raft_consensus.cc:504] T 3d19cf757a874fe28fa14f60b7b48e4c P 3cffed2d98bf4816b9d3d4a263e62b95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:47.175966 16392 leader_election.cc:288] T 3d19cf757a874fe28fa14f60b7b48e4c P 3cffed2d98bf4816b9d3d4a263e62b95 [CANDIDATE]: Term 1 election: Requested vote from peers b725d44cdbe148c2864a6b795ab02019 (127.8.39.130:38975), adc409b2a6554449a338c9c17ca51855 (127.8.39.131:43301)
I20260618 23:24:47.176044 16172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d19cf757a874fe28fa14f60b7b48e4c" candidate_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b725d44cdbe148c2864a6b795ab02019"
I20260618 23:24:47.176096 16312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d19cf757a874fe28fa14f60b7b48e4c" candidate_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc409b2a6554449a338c9c17ca51855"
I20260618 23:24:47.176122 16172 raft_consensus.cc:3037] T 3d19cf757a874fe28fa14f60b7b48e4c P b725d44cdbe148c2864a6b795ab02019 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:47.176164 16312 raft_consensus.cc:3037] T 3d19cf757a874fe28fa14f60b7b48e4c P adc409b2a6554449a338c9c17ca51855 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:47.176800 16312 raft_consensus.cc:2454] T 3d19cf757a874fe28fa14f60b7b48e4c P adc409b2a6554449a338c9c17ca51855 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cffed2d98bf4816b9d3d4a263e62b95 in term 1.
I20260618 23:24:47.176805 16172 raft_consensus.cc:2454] T 3d19cf757a874fe28fa14f60b7b48e4c P b725d44cdbe148c2864a6b795ab02019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cffed2d98bf4816b9d3d4a263e62b95 in term 1.
I20260618 23:24:47.176976 15965 leader_election.cc:302] T 3d19cf757a874fe28fa14f60b7b48e4c P 3cffed2d98bf4816b9d3d4a263e62b95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cffed2d98bf4816b9d3d4a263e62b95, b725d44cdbe148c2864a6b795ab02019; no voters:
I20260618 23:24:47.177079 16392 raft_consensus.cc:2781] T 3d19cf757a874fe28fa14f60b7b48e4c P 3cffed2d98bf4816b9d3d4a263e62b95 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:47.177265 16392 raft_consensus.cc:686] T 3d19cf757a874fe28fa14f60b7b48e4c P 3cffed2d98bf4816b9d3d4a263e62b95 [term 1 LEADER]: Becoming Leader. State: Replica: 3cffed2d98bf4816b9d3d4a263e62b95, State: Running, Role: LEADER
I20260618 23:24:47.177358 16392 consensus_queue.cc:237] T 3d19cf757a874fe28fa14f60b7b48e4c P 3cffed2d98bf4816b9d3d4a263e62b95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:47.178123 15875 catalog_manager.cc:5495] T 3d19cf757a874fe28fa14f60b7b48e4c P 3cffed2d98bf4816b9d3d4a263e62b95 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cffed2d98bf4816b9d3d4a263e62b95 (127.8.39.129). New cstate: current_term: 1 leader_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } health_report { overall_health: UNKNOWN } } }
I20260618 23:24:47.193888 16385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:47.191142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41862 (local address 127.8.39.129:40645)
0618 23:24:47.191204 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:47.191206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:47.191214 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:24:47.191292 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:47.191292 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:47.191308 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:47.191373 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:24:47.191699 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.192148 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.192946 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.193232 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.193290 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:47.193513 (+ 223us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:24:47.193606 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:24:47.193674 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:47.193800 (+ 126us) server_negotiation.cc:299] Negotiation successful
0618 23:24:47.193819 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260618 23:24:47.195623 16312 raft_consensus.cc:1261] T 3d19cf757a874fe28fa14f60b7b48e4c P adc409b2a6554449a338c9c17ca51855 [term 1 FOLLOWER]: Refusing update from remote peer 3cffed2d98bf4816b9d3d4a263e62b95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:24:47.195672 16172 raft_consensus.cc:1261] T 3d19cf757a874fe28fa14f60b7b48e4c P b725d44cdbe148c2864a6b795ab02019 [term 1 FOLLOWER]: Refusing update from remote peer 3cffed2d98bf4816b9d3d4a263e62b95: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:24:47.195969 16392 consensus_queue.cc:1035] T 3d19cf757a874fe28fa14f60b7b48e4c P 3cffed2d98bf4816b9d3d4a263e62b95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:47.196123 16396 consensus_queue.cc:1035] T 3d19cf757a874fe28fa14f60b7b48e4c P 3cffed2d98bf4816b9d3d4a263e62b95 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:47.198297 16399 mvcc.cc:205] Tried to move back new op lower bound from 7298355557150801920 to 7298355557079117824. Current Snapshot: MvccSnapshot[applied={T|T < 7298355557150801920}]
I20260618 23:24:47.198467 16401 mvcc.cc:205] Tried to move back new op lower bound from 7298355557150801920 to 7298355557079117824. Current Snapshot: MvccSnapshot[applied={T|T < 7298355557150801920}]
I20260618 23:24:47.198837 16400 mvcc.cc:205] Tried to move back new op lower bound from 7298355557150801920 to 7298355557079117824. Current Snapshot: MvccSnapshot[applied={T|T < 7298355557150801920}]
I20260618 23:24:47.200627 15874 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:24:47.200722 15874 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:24:47.201490 15874 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71871ac5b89340e7bd6204518a23b6b8: Sending DeleteTablet for 3 replicas of tablet 3d19cf757a874fe28fa14f60b7b48e4c
I20260618 23:24:47.201799 16152 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d19cf757a874fe28fa14f60b7b48e4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:47 UTC) from {username='oryx', principal='oryx/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:53670
I20260618 23:24:47.201916 16012 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d19cf757a874fe28fa14f60b7b48e4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:47 UTC) from {username='oryx', principal='oryx/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:41860
I20260618 23:24:47.201995 16278 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d19cf757a874fe28fa14f60b7b48e4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:47 UTC) from {username='oryx', principal='oryx/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:47048
I20260618 23:24:47.202068 15874 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260618 23:24:47.202188 16410 tablet_replica.cc:330] T 3d19cf757a874fe28fa14f60b7b48e4c P adc409b2a6554449a338c9c17ca51855: stopping tablet replica
W20260618 23:24:47.202212 15874 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:24:47.202266 16410 raft_consensus.cc:2229] T 3d19cf757a874fe28fa14f60b7b48e4c P adc409b2a6554449a338c9c17ca51855 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:24:47.202661 16410 raft_consensus.cc:2258] T 3d19cf757a874fe28fa14f60b7b48e4c P adc409b2a6554449a338c9c17ca51855 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:47.203130 16410 ts_tablet_manager.cc:1905] T 3d19cf757a874fe28fa14f60b7b48e4c P adc409b2a6554449a338c9c17ca51855: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:47.203531 16012 tablet_service.cc:1467] Processing CreateTablet for tablet 3d837daf717d4ccfb8b7845523edd58c (DEFAULT_TABLE table=test-table [id=4aadf7c6a4774856a746ca45c1dfa5f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:47.203641 16012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d837daf717d4ccfb8b7845523edd58c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:47.203830 16152 tablet_service.cc:1467] Processing CreateTablet for tablet 3d837daf717d4ccfb8b7845523edd58c (DEFAULT_TABLE table=test-table [id=4aadf7c6a4774856a746ca45c1dfa5f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:47.203934 16152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d837daf717d4ccfb8b7845523edd58c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:47.204758 16410 ts_tablet_manager.cc:1918] T 3d19cf757a874fe28fa14f60b7b48e4c P adc409b2a6554449a338c9c17ca51855: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:24:47.204820 16410 log.cc:1196] T 3d19cf757a874fe28fa14f60b7b48e4c P adc409b2a6554449a338c9c17ca51855: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-2/wal/wals/3d19cf757a874fe28fa14f60b7b48e4c
I20260618 23:24:47.204975 16386 tablet_bootstrap.cc:492] T 3d837daf717d4ccfb8b7845523edd58c P 3cffed2d98bf4816b9d3d4a263e62b95: Bootstrap starting.
I20260618 23:24:47.205070 16410 ts_tablet_manager.cc:1939] T 3d19cf757a874fe28fa14f60b7b48e4c P adc409b2a6554449a338c9c17ca51855: Deleting consensus metadata
I20260618 23:24:47.205416 16386 tablet_bootstrap.cc:654] T 3d837daf717d4ccfb8b7845523edd58c P 3cffed2d98bf4816b9d3d4a263e62b95: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:47.205482 16388 tablet_bootstrap.cc:492] T 3d837daf717d4ccfb8b7845523edd58c P b725d44cdbe148c2864a6b795ab02019: Bootstrap starting.
I20260618 23:24:47.205631 16408 tablet_replica.cc:330] T 3d19cf757a874fe28fa14f60b7b48e4c P b725d44cdbe148c2864a6b795ab02019: stopping tablet replica
I20260618 23:24:47.205730 16408 raft_consensus.cc:2229] T 3d19cf757a874fe28fa14f60b7b48e4c P b725d44cdbe148c2864a6b795ab02019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:24:47.205832 16408 raft_consensus.cc:2258] T 3d19cf757a874fe28fa14f60b7b48e4c P b725d44cdbe148c2864a6b795ab02019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:47.205931 16388 tablet_bootstrap.cc:654] T 3d837daf717d4ccfb8b7845523edd58c P b725d44cdbe148c2864a6b795ab02019: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:47.205998 16409 tablet_replica.cc:330] T 3d19cf757a874fe28fa14f60b7b48e4c P 3cffed2d98bf4816b9d3d4a263e62b95: stopping tablet replica
I20260618 23:24:47.206094 16409 raft_consensus.cc:2229] T 3d19cf757a874fe28fa14f60b7b48e4c P 3cffed2d98bf4816b9d3d4a263e62b95 [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:24:47.206220 16409 raft_consensus.cc:2258] T 3d19cf757a874fe28fa14f60b7b48e4c P 3cffed2d98bf4816b9d3d4a263e62b95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:47.206347 15859 catalog_manager.cc:4841] TS adc409b2a6554449a338c9c17ca51855 (127.8.39.131:43301): tablet 3d19cf757a874fe28fa14f60b7b48e4c (table test-table [id=1e6824e7d6134a19b73b69aa2ae23d35]) successfully deleted
I20260618 23:24:47.206527 16278 tablet_service.cc:1467] Processing CreateTablet for tablet 3d837daf717d4ccfb8b7845523edd58c (DEFAULT_TABLE table=test-table [id=4aadf7c6a4774856a746ca45c1dfa5f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:47.206624 16278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d837daf717d4ccfb8b7845523edd58c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:47.206749 16409 ts_tablet_manager.cc:1905] T 3d19cf757a874fe28fa14f60b7b48e4c P 3cffed2d98bf4816b9d3d4a263e62b95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:47.207818 16387 tablet_bootstrap.cc:492] T 3d837daf717d4ccfb8b7845523edd58c P adc409b2a6554449a338c9c17ca51855: Bootstrap starting.
I20260618 23:24:47.208205 16387 tablet_bootstrap.cc:654] T 3d837daf717d4ccfb8b7845523edd58c P adc409b2a6554449a338c9c17ca51855: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:47.208340 16388 tablet_bootstrap.cc:492] T 3d837daf717d4ccfb8b7845523edd58c P b725d44cdbe148c2864a6b795ab02019: No bootstrap required, opened a new log
I20260618 23:24:47.208384 16388 ts_tablet_manager.cc:1397] T 3d837daf717d4ccfb8b7845523edd58c P b725d44cdbe148c2864a6b795ab02019: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260618 23:24:47.208518 16388 raft_consensus.cc:348] T 3d837daf717d4ccfb8b7845523edd58c P b725d44cdbe148c2864a6b795ab02019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:47.208577 16388 raft_consensus.cc:374] T 3d837daf717d4ccfb8b7845523edd58c P b725d44cdbe148c2864a6b795ab02019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:47.208597 16388 raft_consensus.cc:729] T 3d837daf717d4ccfb8b7845523edd58c P b725d44cdbe148c2864a6b795ab02019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b725d44cdbe148c2864a6b795ab02019, State: Initialized, Role: FOLLOWER
I20260618 23:24:47.208640 16388 consensus_queue.cc:260] T 3d837daf717d4ccfb8b7845523edd58c P b725d44cdbe148c2864a6b795ab02019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:47.208722 16387 tablet_bootstrap.cc:492] T 3d837daf717d4ccfb8b7845523edd58c P adc409b2a6554449a338c9c17ca51855: No bootstrap required, opened a new log
I20260618 23:24:47.208734 16388 ts_tablet_manager.cc:1428] T 3d837daf717d4ccfb8b7845523edd58c P b725d44cdbe148c2864a6b795ab02019: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:47.208760 16387 ts_tablet_manager.cc:1397] T 3d837daf717d4ccfb8b7845523edd58c P adc409b2a6554449a338c9c17ca51855: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:47.208917 16409 ts_tablet_manager.cc:1918] T 3d19cf757a874fe28fa14f60b7b48e4c P 3cffed2d98bf4816b9d3d4a263e62b95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:24:47.208912 16387 raft_consensus.cc:348] T 3d837daf717d4ccfb8b7845523edd58c P adc409b2a6554449a338c9c17ca51855 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:47.208984 16409 log.cc:1196] T 3d19cf757a874fe28fa14f60b7b48e4c P 3cffed2d98bf4816b9d3d4a263e62b95: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-0/wal/wals/3d19cf757a874fe28fa14f60b7b48e4c
I20260618 23:24:47.208982 16387 raft_consensus.cc:374] T 3d837daf717d4ccfb8b7845523edd58c P adc409b2a6554449a338c9c17ca51855 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:47.209012 16387 raft_consensus.cc:729] T 3d837daf717d4ccfb8b7845523edd58c P adc409b2a6554449a338c9c17ca51855 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adc409b2a6554449a338c9c17ca51855, State: Initialized, Role: FOLLOWER
I20260618 23:24:47.209065 16387 consensus_queue.cc:260] T 3d837daf717d4ccfb8b7845523edd58c P adc409b2a6554449a338c9c17ca51855 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:47.209190 16387 ts_tablet_manager.cc:1428] T 3d837daf717d4ccfb8b7845523edd58c P adc409b2a6554449a338c9c17ca51855: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260618 23:24:47.209264 16409 ts_tablet_manager.cc:1939] T 3d19cf757a874fe28fa14f60b7b48e4c P 3cffed2d98bf4816b9d3d4a263e62b95: Deleting consensus metadata
I20260618 23:24:47.209430 16408 ts_tablet_manager.cc:1905] T 3d19cf757a874fe28fa14f60b7b48e4c P b725d44cdbe148c2864a6b795ab02019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:47.209594 15860 catalog_manager.cc:4841] TS 3cffed2d98bf4816b9d3d4a263e62b95 (127.8.39.129:40645): tablet 3d19cf757a874fe28fa14f60b7b48e4c (table test-table [id=1e6824e7d6134a19b73b69aa2ae23d35]) successfully deleted
I20260618 23:24:47.210691 16408 ts_tablet_manager.cc:1918] T 3d19cf757a874fe28fa14f60b7b48e4c P b725d44cdbe148c2864a6b795ab02019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:24:47.210747 16408 log.cc:1196] T 3d19cf757a874fe28fa14f60b7b48e4c P b725d44cdbe148c2864a6b795ab02019: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-1/wal/wals/3d19cf757a874fe28fa14f60b7b48e4c
I20260618 23:24:47.211091 16408 ts_tablet_manager.cc:1939] T 3d19cf757a874fe28fa14f60b7b48e4c P b725d44cdbe148c2864a6b795ab02019: Deleting consensus metadata
I20260618 23:24:47.211476 15861 catalog_manager.cc:4841] TS b725d44cdbe148c2864a6b795ab02019 (127.8.39.130:38975): tablet 3d19cf757a874fe28fa14f60b7b48e4c (table test-table [id=1e6824e7d6134a19b73b69aa2ae23d35]) successfully deleted
I20260618 23:24:47.213582 16386 tablet_bootstrap.cc:492] T 3d837daf717d4ccfb8b7845523edd58c P 3cffed2d98bf4816b9d3d4a263e62b95: No bootstrap required, opened a new log
I20260618 23:24:47.213641 16386 ts_tablet_manager.cc:1397] T 3d837daf717d4ccfb8b7845523edd58c P 3cffed2d98bf4816b9d3d4a263e62b95: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20260618 23:24:47.213778 16386 raft_consensus.cc:348] T 3d837daf717d4ccfb8b7845523edd58c P 3cffed2d98bf4816b9d3d4a263e62b95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:47.213887 16386 raft_consensus.cc:374] T 3d837daf717d4ccfb8b7845523edd58c P 3cffed2d98bf4816b9d3d4a263e62b95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:47.213909 16386 raft_consensus.cc:729] T 3d837daf717d4ccfb8b7845523edd58c P 3cffed2d98bf4816b9d3d4a263e62b95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cffed2d98bf4816b9d3d4a263e62b95, State: Initialized, Role: FOLLOWER
I20260618 23:24:47.213950 16386 consensus_queue.cc:260] T 3d837daf717d4ccfb8b7845523edd58c P 3cffed2d98bf4816b9d3d4a263e62b95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:47.214031 16386 ts_tablet_manager.cc:1428] T 3d837daf717d4ccfb8b7845523edd58c P 3cffed2d98bf4816b9d3d4a263e62b95: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260618 23:24:47.238826 16394 raft_consensus.cc:482] T 3d837daf717d4ccfb8b7845523edd58c P b725d44cdbe148c2864a6b795ab02019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:24:47.238922 16394 raft_consensus.cc:504] T 3d837daf717d4ccfb8b7845523edd58c P b725d44cdbe148c2864a6b795ab02019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:47.239281 16394 leader_election.cc:288] T 3d837daf717d4ccfb8b7845523edd58c P b725d44cdbe148c2864a6b795ab02019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cffed2d98bf4816b9d3d4a263e62b95 (127.8.39.129:40645), adc409b2a6554449a338c9c17ca51855 (127.8.39.131:43301)
I20260618 23:24:47.242838 16385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:47.239710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:34385 (local address 127.8.39.129:40645)
0618 23:24:47.239752 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:47.239754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:47.239870 (+ 116us) server_negotiation.cc:407] Connection header received
0618 23:24:47.239937 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:47.239938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:47.239947 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:47.240009 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:47.240151 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.240920 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.242089 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.242458 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.242545 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:47.242554 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:47.242719 (+ 165us) server_negotiation.cc:299] Negotiation successful
0618 23:24:47.242759 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260618 23:24:47.242918 16111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:47.239460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:43301 (local address 127.8.39.130:34975)
0618 23:24:47.239515 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:47.239527 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:47.239579 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:47.239859 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:47.239860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:47.239862 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:47.239933 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:47.239935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.240617 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.240618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:47.241713 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:47.241716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.242681 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.242682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:47.242751 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:47.242753 (+ 2us) client_negotiation.cc:769] Sending connection context
0618 23:24:47.242806 (+ 53us) client_negotiation.cc:240] Negotiation successful
0618 23:24:47.242843 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260618 23:24:47.242949 16384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:47.239670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:34975 (local address 127.8.39.131:43301)
0618 23:24:47.239711 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:47.239713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:47.239721 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:24:47.239748 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:47.239748 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:47.239759 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:47.239809 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:47.239990 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.240523 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.241865 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.242591 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.242678 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:47.242688 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:47.242826 (+ 138us) server_negotiation.cc:299] Negotiation successful
0618 23:24:47.242869 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260618 23:24:47.243170 16414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:47.239663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:40645 (local address 127.8.39.130:34385)
0618 23:24:47.239800 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:47.239810 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:47.239878 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:47.240016 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:47.240017 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:47.240018 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:47.240084 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:47.240085 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.241016 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.241017 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:47.242004 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:47.242006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.242544 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.242544 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:47.242613 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:47.242617 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:24:47.242960 (+ 343us) client_negotiation.cc:240] Negotiation successful
0618 23:24:47.243103 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
I20260618 23:24:47.243166 16312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d837daf717d4ccfb8b7845523edd58c" candidate_uuid: "b725d44cdbe148c2864a6b795ab02019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc409b2a6554449a338c9c17ca51855" is_pre_election: true
I20260618 23:24:47.243253 16312 raft_consensus.cc:2454] T 3d837daf717d4ccfb8b7845523edd58c P adc409b2a6554449a338c9c17ca51855 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b725d44cdbe148c2864a6b795ab02019 in term 0.
I20260618 23:24:47.243474 16103 leader_election.cc:302] T 3d837daf717d4ccfb8b7845523edd58c P b725d44cdbe148c2864a6b795ab02019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adc409b2a6554449a338c9c17ca51855, b725d44cdbe148c2864a6b795ab02019; no voters:
I20260618 23:24:47.243584 16394 raft_consensus.cc:2781] T 3d837daf717d4ccfb8b7845523edd58c P b725d44cdbe148c2864a6b795ab02019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:24:47.243626 16394 raft_consensus.cc:482] T 3d837daf717d4ccfb8b7845523edd58c P b725d44cdbe148c2864a6b795ab02019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:24:47.243650 16394 raft_consensus.cc:3037] T 3d837daf717d4ccfb8b7845523edd58c P b725d44cdbe148c2864a6b795ab02019 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:47.243637 16032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d837daf717d4ccfb8b7845523edd58c" candidate_uuid: "b725d44cdbe148c2864a6b795ab02019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" is_pre_election: true
I20260618 23:24:47.243744 16032 raft_consensus.cc:2454] T 3d837daf717d4ccfb8b7845523edd58c P 3cffed2d98bf4816b9d3d4a263e62b95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b725d44cdbe148c2864a6b795ab02019 in term 0.
I20260618 23:24:47.244413 16394 raft_consensus.cc:504] T 3d837daf717d4ccfb8b7845523edd58c P b725d44cdbe148c2864a6b795ab02019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:47.244550 16394 leader_election.cc:288] T 3d837daf717d4ccfb8b7845523edd58c P b725d44cdbe148c2864a6b795ab02019 [CANDIDATE]: Term 1 election: Requested vote from peers 3cffed2d98bf4816b9d3d4a263e62b95 (127.8.39.129:40645), adc409b2a6554449a338c9c17ca51855 (127.8.39.131:43301)
I20260618 23:24:47.244691 16032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d837daf717d4ccfb8b7845523edd58c" candidate_uuid: "b725d44cdbe148c2864a6b795ab02019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cffed2d98bf4816b9d3d4a263e62b95"
I20260618 23:24:47.244733 16312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d837daf717d4ccfb8b7845523edd58c" candidate_uuid: "b725d44cdbe148c2864a6b795ab02019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc409b2a6554449a338c9c17ca51855"
I20260618 23:24:47.244772 16032 raft_consensus.cc:3037] T 3d837daf717d4ccfb8b7845523edd58c P 3cffed2d98bf4816b9d3d4a263e62b95 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:47.244807 16312 raft_consensus.cc:3037] T 3d837daf717d4ccfb8b7845523edd58c P adc409b2a6554449a338c9c17ca51855 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:47.245463 16312 raft_consensus.cc:2454] T 3d837daf717d4ccfb8b7845523edd58c P adc409b2a6554449a338c9c17ca51855 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b725d44cdbe148c2864a6b795ab02019 in term 1.
I20260618 23:24:47.245591 16032 raft_consensus.cc:2454] T 3d837daf717d4ccfb8b7845523edd58c P 3cffed2d98bf4816b9d3d4a263e62b95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b725d44cdbe148c2864a6b795ab02019 in term 1.
I20260618 23:24:47.245620 16103 leader_election.cc:302] T 3d837daf717d4ccfb8b7845523edd58c P b725d44cdbe148c2864a6b795ab02019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adc409b2a6554449a338c9c17ca51855, b725d44cdbe148c2864a6b795ab02019; no voters:
I20260618 23:24:47.245697 16394 raft_consensus.cc:2781] T 3d837daf717d4ccfb8b7845523edd58c P b725d44cdbe148c2864a6b795ab02019 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:47.245829 16394 raft_consensus.cc:686] T 3d837daf717d4ccfb8b7845523edd58c P b725d44cdbe148c2864a6b795ab02019 [term 1 LEADER]: Becoming Leader. State: Replica: b725d44cdbe148c2864a6b795ab02019, State: Running, Role: LEADER
I20260618 23:24:47.245904 16394 consensus_queue.cc:237] T 3d837daf717d4ccfb8b7845523edd58c P b725d44cdbe148c2864a6b795ab02019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:47.246546 15874 catalog_manager.cc:5495] T 3d837daf717d4ccfb8b7845523edd58c P b725d44cdbe148c2864a6b795ab02019 reported cstate change: term changed from 0 to 1, leader changed from <none> to b725d44cdbe148c2864a6b795ab02019 (127.8.39.130). New cstate: current_term: 1 leader_uuid: "b725d44cdbe148c2864a6b795ab02019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } health_report { overall_health: UNKNOWN } } }
W20260618 23:24:47.264109 16079 tablet.cc:2369] T 3d837daf717d4ccfb8b7845523edd58c P 3cffed2d98bf4816b9d3d4a263e62b95: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 18 23:24:47 dist-test-slave-47qt krb5kdc[15830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825086, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.190@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM
I20260618 23:24:47.284885 15924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:47.277934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38036 (local address 127.8.39.190:44111)
0618 23:24:47.277988 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:47.277990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:47.278093 (+ 103us) server_negotiation.cc:407] Connection header received
0618 23:24:47.278172 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:47.278173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:47.278182 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:47.278263 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:47.278699 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.279269 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.279988 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.280179 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.280260 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:47.282291 (+ 2031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:47.282292 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:47.282293 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:47.282299 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:47.283701 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:47.284088 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:47.284089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:47.284090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:47.284146 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:47.284344 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:47.284345 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:47.284346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:47.284452 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:47.284526 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:47.284670 (+ 144us) server_negotiation.cc:299] Negotiation successful
0618 23:24:47.284750 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260618 23:24:47.284953 16422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:47.277867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:44111 (local address 127.0.0.1:38036)
0618 23:24:47.278013 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:47.278030 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:47.278112 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:47.278268 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:47.278269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:47.278564 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:47.278608 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:47.278610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.279368 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.279369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:47.279891 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:47.279892 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.280260 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.280261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:47.280310 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:47.280853 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:47.280857 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:47.282201 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:47.283802 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:47.283809 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:47.283811 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:47.284000 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:47.284214 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:47.284215 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:47.284215 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:47.284272 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:47.284515 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:47.284522 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:47.284563 (+ 41us) client_negotiation.cc:769] Sending connection context
0618 23:24:47.284650 (+ 87us) client_negotiation.cc:240] Negotiation successful
0618 23:24:47.284801 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":59,"thread_start_us":75,"threads_started":1}
I20260618 23:24:47.287300 15874 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:38036:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260618 23:24:47.287526 15874 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:24:47.288962 16012 tablet_service.cc:1467] Processing CreateTablet for tablet c781f17aae5d4dd3bc9955b448d52d18 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f257846e8f514106aa467eabdcff5d70]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 23:24:47.288975 16152 tablet_service.cc:1467] Processing CreateTablet for tablet c781f17aae5d4dd3bc9955b448d52d18 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f257846e8f514106aa467eabdcff5d70]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 23:24:47.289141 16012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c781f17aae5d4dd3bc9955b448d52d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:47.289156 16152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c781f17aae5d4dd3bc9955b448d52d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:47.290215 16388 tablet_bootstrap.cc:492] T c781f17aae5d4dd3bc9955b448d52d18 P b725d44cdbe148c2864a6b795ab02019: Bootstrap starting.
I20260618 23:24:47.290316 16278 tablet_service.cc:1467] Processing CreateTablet for tablet c781f17aae5d4dd3bc9955b448d52d18 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f257846e8f514106aa467eabdcff5d70]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260618 23:24:47.290469 16278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c781f17aae5d4dd3bc9955b448d52d18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:47.290642 16388 tablet_bootstrap.cc:654] T c781f17aae5d4dd3bc9955b448d52d18 P b725d44cdbe148c2864a6b795ab02019: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:47.290695 16386 tablet_bootstrap.cc:492] T c781f17aae5d4dd3bc9955b448d52d18 P 3cffed2d98bf4816b9d3d4a263e62b95: Bootstrap starting.
I20260618 23:24:47.291095 16386 tablet_bootstrap.cc:654] T c781f17aae5d4dd3bc9955b448d52d18 P 3cffed2d98bf4816b9d3d4a263e62b95: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:47.291803 16387 tablet_bootstrap.cc:492] T c781f17aae5d4dd3bc9955b448d52d18 P adc409b2a6554449a338c9c17ca51855: Bootstrap starting.
I20260618 23:24:47.291999 16386 tablet_bootstrap.cc:492] T c781f17aae5d4dd3bc9955b448d52d18 P 3cffed2d98bf4816b9d3d4a263e62b95: No bootstrap required, opened a new log
I20260618 23:24:47.292055 16386 ts_tablet_manager.cc:1397] T c781f17aae5d4dd3bc9955b448d52d18 P 3cffed2d98bf4816b9d3d4a263e62b95: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260618 23:24:47.292204 16387 tablet_bootstrap.cc:654] T c781f17aae5d4dd3bc9955b448d52d18 P adc409b2a6554449a338c9c17ca51855: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:47.292198 16386 raft_consensus.cc:348] T c781f17aae5d4dd3bc9955b448d52d18 P 3cffed2d98bf4816b9d3d4a263e62b95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:47.292268 16386 raft_consensus.cc:374] T c781f17aae5d4dd3bc9955b448d52d18 P 3cffed2d98bf4816b9d3d4a263e62b95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:47.292296 16386 raft_consensus.cc:729] T c781f17aae5d4dd3bc9955b448d52d18 P 3cffed2d98bf4816b9d3d4a263e62b95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cffed2d98bf4816b9d3d4a263e62b95, State: Initialized, Role: FOLLOWER
I20260618 23:24:47.292338 16386 consensus_queue.cc:260] T c781f17aae5d4dd3bc9955b448d52d18 P 3cffed2d98bf4816b9d3d4a263e62b95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:47.292426 16386 ts_tablet_manager.cc:1428] T c781f17aae5d4dd3bc9955b448d52d18 P 3cffed2d98bf4816b9d3d4a263e62b95: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260618 23:24:47.292529 16392 tablet_replica.cc:439] T c781f17aae5d4dd3bc9955b448d52d18 P 3cffed2d98bf4816b9d3d4a263e62b95: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } } }
I20260618 23:24:47.292596 16392 tablet_replica.cc:442] T c781f17aae5d4dd3bc9955b448d52d18 P 3cffed2d98bf4816b9d3d4a263e62b95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:24:47.292971 16387 tablet_bootstrap.cc:492] T c781f17aae5d4dd3bc9955b448d52d18 P adc409b2a6554449a338c9c17ca51855: No bootstrap required, opened a new log
I20260618 23:24:47.293027 16387 ts_tablet_manager.cc:1397] T c781f17aae5d4dd3bc9955b448d52d18 P adc409b2a6554449a338c9c17ca51855: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:47.293190 16387 raft_consensus.cc:348] T c781f17aae5d4dd3bc9955b448d52d18 P adc409b2a6554449a338c9c17ca51855 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:47.293289 16387 raft_consensus.cc:374] T c781f17aae5d4dd3bc9955b448d52d18 P adc409b2a6554449a338c9c17ca51855 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:47.293514 16387 raft_consensus.cc:729] T c781f17aae5d4dd3bc9955b448d52d18 P adc409b2a6554449a338c9c17ca51855 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adc409b2a6554449a338c9c17ca51855, State: Initialized, Role: FOLLOWER
I20260618 23:24:47.293614 16387 consensus_queue.cc:260] T c781f17aae5d4dd3bc9955b448d52d18 P adc409b2a6554449a338c9c17ca51855 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:47.293812 16387 ts_tablet_manager.cc:1428] T c781f17aae5d4dd3bc9955b448d52d18 P adc409b2a6554449a338c9c17ca51855: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260618 23:24:47.293864 16393 tablet_replica.cc:439] T c781f17aae5d4dd3bc9955b448d52d18 P adc409b2a6554449a338c9c17ca51855: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } } }
I20260618 23:24:47.293998 16393 tablet_replica.cc:442] T c781f17aae5d4dd3bc9955b448d52d18 P adc409b2a6554449a338c9c17ca51855: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:24:47.294653 16388 tablet_bootstrap.cc:492] T c781f17aae5d4dd3bc9955b448d52d18 P b725d44cdbe148c2864a6b795ab02019: No bootstrap required, opened a new log
I20260618 23:24:47.294727 16388 ts_tablet_manager.cc:1397] T c781f17aae5d4dd3bc9955b448d52d18 P b725d44cdbe148c2864a6b795ab02019: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260618 23:24:47.294907 16388 raft_consensus.cc:348] T c781f17aae5d4dd3bc9955b448d52d18 P b725d44cdbe148c2864a6b795ab02019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:47.295014 16388 raft_consensus.cc:374] T c781f17aae5d4dd3bc9955b448d52d18 P b725d44cdbe148c2864a6b795ab02019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:47.295040 16388 raft_consensus.cc:729] T c781f17aae5d4dd3bc9955b448d52d18 P b725d44cdbe148c2864a6b795ab02019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b725d44cdbe148c2864a6b795ab02019, State: Initialized, Role: FOLLOWER
I20260618 23:24:47.295089 16388 consensus_queue.cc:260] T c781f17aae5d4dd3bc9955b448d52d18 P b725d44cdbe148c2864a6b795ab02019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:47.295176 16388 ts_tablet_manager.cc:1428] T c781f17aae5d4dd3bc9955b448d52d18 P b725d44cdbe148c2864a6b795ab02019: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260618 23:24:47.295293 16394 tablet_replica.cc:439] T c781f17aae5d4dd3bc9955b448d52d18 P b725d44cdbe148c2864a6b795ab02019: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } } }
I20260618 23:24:47.295377 16394 tablet_replica.cc:442] T c781f17aae5d4dd3bc9955b448d52d18 P b725d44cdbe148c2864a6b795ab02019: This TxnStatusTablet replica's current role is: FOLLOWER
W20260618 23:24:47.319275 16359 tablet.cc:2369] T 3d837daf717d4ccfb8b7845523edd58c P adc409b2a6554449a338c9c17ca51855: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:24:47.597420 16394 raft_consensus.cc:482] T c781f17aae5d4dd3bc9955b448d52d18 P b725d44cdbe148c2864a6b795ab02019 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:24:47.597536 16394 raft_consensus.cc:504] T c781f17aae5d4dd3bc9955b448d52d18 P b725d44cdbe148c2864a6b795ab02019 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:47.597739 16394 leader_election.cc:288] T c781f17aae5d4dd3bc9955b448d52d18 P b725d44cdbe148c2864a6b795ab02019 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cffed2d98bf4816b9d3d4a263e62b95 (127.8.39.129:40645), adc409b2a6554449a338c9c17ca51855 (127.8.39.131:43301)
I20260618 23:24:47.598049 16312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c781f17aae5d4dd3bc9955b448d52d18" candidate_uuid: "b725d44cdbe148c2864a6b795ab02019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc409b2a6554449a338c9c17ca51855" is_pre_election: true
I20260618 23:24:47.598104 16032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c781f17aae5d4dd3bc9955b448d52d18" candidate_uuid: "b725d44cdbe148c2864a6b795ab02019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" is_pre_election: true
I20260618 23:24:47.598179 16312 raft_consensus.cc:2454] T c781f17aae5d4dd3bc9955b448d52d18 P adc409b2a6554449a338c9c17ca51855 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b725d44cdbe148c2864a6b795ab02019 in term 0.
I20260618 23:24:47.598188 16032 raft_consensus.cc:2454] T c781f17aae5d4dd3bc9955b448d52d18 P 3cffed2d98bf4816b9d3d4a263e62b95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b725d44cdbe148c2864a6b795ab02019 in term 0.
I20260618 23:24:47.598384 16104 leader_election.cc:302] T c781f17aae5d4dd3bc9955b448d52d18 P b725d44cdbe148c2864a6b795ab02019 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cffed2d98bf4816b9d3d4a263e62b95, b725d44cdbe148c2864a6b795ab02019; no voters:
I20260618 23:24:47.598492 16394 raft_consensus.cc:2781] T c781f17aae5d4dd3bc9955b448d52d18 P b725d44cdbe148c2864a6b795ab02019 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:24:47.598536 16394 raft_consensus.cc:482] T c781f17aae5d4dd3bc9955b448d52d18 P b725d44cdbe148c2864a6b795ab02019 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:24:47.598554 16394 raft_consensus.cc:3037] T c781f17aae5d4dd3bc9955b448d52d18 P b725d44cdbe148c2864a6b795ab02019 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:47.599529 16394 raft_consensus.cc:504] T c781f17aae5d4dd3bc9955b448d52d18 P b725d44cdbe148c2864a6b795ab02019 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:47.599678 16394 leader_election.cc:288] T c781f17aae5d4dd3bc9955b448d52d18 P b725d44cdbe148c2864a6b795ab02019 [CANDIDATE]: Term 1 election: Requested vote from peers 3cffed2d98bf4816b9d3d4a263e62b95 (127.8.39.129:40645), adc409b2a6554449a338c9c17ca51855 (127.8.39.131:43301)
I20260618 23:24:47.599854 16032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c781f17aae5d4dd3bc9955b448d52d18" candidate_uuid: "b725d44cdbe148c2864a6b795ab02019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cffed2d98bf4816b9d3d4a263e62b95"
I20260618 23:24:47.599927 16032 raft_consensus.cc:3037] T c781f17aae5d4dd3bc9955b448d52d18 P 3cffed2d98bf4816b9d3d4a263e62b95 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:47.599972 16312 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c781f17aae5d4dd3bc9955b448d52d18" candidate_uuid: "b725d44cdbe148c2864a6b795ab02019" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "adc409b2a6554449a338c9c17ca51855"
I20260618 23:24:47.600057 16312 raft_consensus.cc:3037] T c781f17aae5d4dd3bc9955b448d52d18 P adc409b2a6554449a338c9c17ca51855 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:47.600692 16032 raft_consensus.cc:2454] T c781f17aae5d4dd3bc9955b448d52d18 P 3cffed2d98bf4816b9d3d4a263e62b95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b725d44cdbe148c2864a6b795ab02019 in term 1.
I20260618 23:24:47.600728 16312 raft_consensus.cc:2454] T c781f17aae5d4dd3bc9955b448d52d18 P adc409b2a6554449a338c9c17ca51855 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b725d44cdbe148c2864a6b795ab02019 in term 1.
I20260618 23:24:47.600880 16103 leader_election.cc:302] T c781f17aae5d4dd3bc9955b448d52d18 P b725d44cdbe148c2864a6b795ab02019 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adc409b2a6554449a338c9c17ca51855, b725d44cdbe148c2864a6b795ab02019; no voters:
I20260618 23:24:47.600991 16394 raft_consensus.cc:2781] T c781f17aae5d4dd3bc9955b448d52d18 P b725d44cdbe148c2864a6b795ab02019 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:47.601060 16394 raft_consensus.cc:686] T c781f17aae5d4dd3bc9955b448d52d18 P b725d44cdbe148c2864a6b795ab02019 [term 1 LEADER]: Becoming Leader. State: Replica: b725d44cdbe148c2864a6b795ab02019, State: Running, Role: LEADER
I20260618 23:24:47.601140 16394 consensus_queue.cc:237] T c781f17aae5d4dd3bc9955b448d52d18 P b725d44cdbe148c2864a6b795ab02019 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:47.601418 16415 tablet_replica.cc:439] T c781f17aae5d4dd3bc9955b448d52d18 P b725d44cdbe148c2864a6b795ab02019: TxnStatusTablet state changed. Reason: New leader b725d44cdbe148c2864a6b795ab02019. Latest consensus state: current_term: 1 leader_uuid: "b725d44cdbe148c2864a6b795ab02019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } } }
I20260618 23:24:47.601486 16415 tablet_replica.cc:442] T c781f17aae5d4dd3bc9955b448d52d18 P b725d44cdbe148c2864a6b795ab02019: This TxnStatusTablet replica's current role is: LEADER
I20260618 23:24:47.601701 16427 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260618 23:24:47.601737 15874 catalog_manager.cc:5495] T c781f17aae5d4dd3bc9955b448d52d18 P b725d44cdbe148c2864a6b795ab02019 reported cstate change: term changed from 0 to 1, leader changed from <none> to b725d44cdbe148c2864a6b795ab02019 (127.8.39.130). New cstate: current_term: 1 leader_uuid: "b725d44cdbe148c2864a6b795ab02019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } health_report { overall_health: UNKNOWN } } }
I20260618 23:24:47.601868 16427 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260618 23:24:47.649509 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:47.646676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53672 (local address 127.8.39.130:38975)
0618 23:24:47.646717 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:47.646719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:47.646762 (+ 43us) server_negotiation.cc:407] Connection header received
0618 23:24:47.646856 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:47.646857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:47.646872 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:47.646922 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:24:47.647063 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.647648 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.648747 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.648923 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.648968 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:47.649193 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:24:47.649299 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:24:47.649355 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:47.649438 (+ 83us) server_negotiation.cc:299] Negotiation successful
0618 23:24:47.649454 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260618 23:24:47.649564 16422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:47.646616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:38975 (local address 127.0.0.1:53672)
0618 23:24:47.646707 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:47.646720 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:47.646779 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:47.646934 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:47.646935 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:47.646937 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 23:24:47.646987 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:47.646989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.647733 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.647734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:47.648660 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:47.648663 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.648988 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.648989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:47.649057 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:47.649072 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 23:24:47.649371 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 23:24:47.649378 (+ 7us) client_negotiation.cc:769] Sending connection context
0618 23:24:47.649435 (+ 57us) client_negotiation.cc:240] Negotiation successful
0618 23:24:47.649483 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260618 23:24:47.650660 16312 raft_consensus.cc:1261] T c781f17aae5d4dd3bc9955b448d52d18 P adc409b2a6554449a338c9c17ca51855 [term 1 FOLLOWER]: Refusing update from remote peer b725d44cdbe148c2864a6b795ab02019: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:24:47.650799 16032 raft_consensus.cc:1261] T c781f17aae5d4dd3bc9955b448d52d18 P 3cffed2d98bf4816b9d3d4a263e62b95 [term 1 FOLLOWER]: Refusing update from remote peer b725d44cdbe148c2864a6b795ab02019: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:24:47.651121 16415 consensus_queue.cc:1035] T c781f17aae5d4dd3bc9955b448d52d18 P b725d44cdbe148c2864a6b795ab02019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:47.651283 16394 consensus_queue.cc:1035] T c781f17aae5d4dd3bc9955b448d52d18 P b725d44cdbe148c2864a6b795ab02019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:47.651891 16392 tablet_replica.cc:439] T c781f17aae5d4dd3bc9955b448d52d18 P 3cffed2d98bf4816b9d3d4a263e62b95: TxnStatusTablet state changed. Reason: New leader b725d44cdbe148c2864a6b795ab02019. Latest consensus state: current_term: 1 leader_uuid: "b725d44cdbe148c2864a6b795ab02019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } } }
I20260618 23:24:47.651991 16392 tablet_replica.cc:442] T c781f17aae5d4dd3bc9955b448d52d18 P 3cffed2d98bf4816b9d3d4a263e62b95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:24:47.652076 16393 tablet_replica.cc:439] T c781f17aae5d4dd3bc9955b448d52d18 P adc409b2a6554449a338c9c17ca51855: TxnStatusTablet state changed. Reason: New leader b725d44cdbe148c2864a6b795ab02019. Latest consensus state: current_term: 1 leader_uuid: "b725d44cdbe148c2864a6b795ab02019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } } }
I20260618 23:24:47.652172 16393 tablet_replica.cc:442] T c781f17aae5d4dd3bc9955b448d52d18 P adc409b2a6554449a338c9c17ca51855: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:24:47.652633 16394 tablet_replica.cc:439] T c781f17aae5d4dd3bc9955b448d52d18 P b725d44cdbe148c2864a6b795ab02019: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b725d44cdbe148c2864a6b795ab02019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } } }
I20260618 23:24:47.652715 16394 tablet_replica.cc:442] T c781f17aae5d4dd3bc9955b448d52d18 P b725d44cdbe148c2864a6b795ab02019: This TxnStatusTablet replica's current role is: LEADER
I20260618 23:24:47.653247 16415 tablet_replica.cc:439] T c781f17aae5d4dd3bc9955b448d52d18 P b725d44cdbe148c2864a6b795ab02019: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b725d44cdbe148c2864a6b795ab02019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } } }
I20260618 23:24:47.653322 16415 tablet_replica.cc:442] T c781f17aae5d4dd3bc9955b448d52d18 P b725d44cdbe148c2864a6b795ab02019: This TxnStatusTablet replica's current role is: LEADER
I20260618 23:24:47.653555 16392 tablet_replica.cc:439] T c781f17aae5d4dd3bc9955b448d52d18 P 3cffed2d98bf4816b9d3d4a263e62b95: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b725d44cdbe148c2864a6b795ab02019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } } }
I20260618 23:24:47.653638 16392 tablet_replica.cc:442] T c781f17aae5d4dd3bc9955b448d52d18 P 3cffed2d98bf4816b9d3d4a263e62b95: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:24:47.654170 16393 tablet_replica.cc:439] T c781f17aae5d4dd3bc9955b448d52d18 P adc409b2a6554449a338c9c17ca51855: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b725d44cdbe148c2864a6b795ab02019" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } } }
I20260618 23:24:47.654254 16393 tablet_replica.cc:442] T c781f17aae5d4dd3bc9955b448d52d18 P adc409b2a6554449a338c9c17ca51855: This TxnStatusTablet replica's current role is: FOLLOWER
I20260618 23:24:47.657238 16394 consensus_queue.cc:1035] T 3d837daf717d4ccfb8b7845523edd58c P b725d44cdbe148c2864a6b795ab02019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:47.658270 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:47.655283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53688 (local address 127.8.39.130:38975)
0618 23:24:47.655338 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:47.655340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:47.655348 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:24:47.655369 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:47.655369 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:47.655379 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:47.655457 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:24:47.655580 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.656185 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.657281 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.657607 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.657676 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:47.658007 (+ 331us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:24:47.658042 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:24:47.658093 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:47.658177 (+ 84us) server_negotiation.cc:299] Negotiation successful
0618 23:24:47.658198 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260618 23:24:47.658541 16415 consensus_queue.cc:1035] T 3d837daf717d4ccfb8b7845523edd58c P b725d44cdbe148c2864a6b795ab02019 [LEADER]: Connected to new peer: Peer: permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:47.663808 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:47.661196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:56625 (local address 127.8.39.130:38975)
0618 23:24:47.661252 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:47.661254 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:47.661362 (+ 108us) server_negotiation.cc:407] Connection header received
0618 23:24:47.661385 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:47.661386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:47.661394 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:47.661456 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:24:47.661544 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.662255 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.662976 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.663269 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.663343 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:47.663512 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:24:47.663543 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:24:47.663598 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:47.663701 (+ 103us) server_negotiation.cc:299] Negotiation successful
0618 23:24:47.663717 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260618 23:24:47.663861 16436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:47.661100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:38975 (local address 127.8.39.130:56625)
0618 23:24:47.661263 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:47.661273 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:47.661351 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:47.661474 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:47.661475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:47.661476 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 23:24:47.661507 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:47.661508 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.662346 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.662346 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:47.662893 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:47.662895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.663365 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.663366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:47.663414 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:47.663427 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 23:24:47.663631 (+ 204us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 23:24:47.663634 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:24:47.663691 (+ 57us) client_negotiation.cc:240] Negotiation successful
0618 23:24:47.663737 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":103,"threads_started":1}
I20260618 23:24:47.677050 15875 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:24:47.677174 15875 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:24:47.677850 15875 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71871ac5b89340e7bd6204518a23b6b8: Sending DeleteTablet for 3 replicas of tablet 3d837daf717d4ccfb8b7845523edd58c
I20260618 23:24:47.678069 16278 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d837daf717d4ccfb8b7845523edd58c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:47 UTC) from {username='oryx', principal='oryx/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:47048
I20260618 23:24:47.678184 16410 tablet_replica.cc:330] T 3d837daf717d4ccfb8b7845523edd58c P adc409b2a6554449a338c9c17ca51855: stopping tablet replica
I20260618 23:24:47.678200 16152 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d837daf717d4ccfb8b7845523edd58c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:47 UTC) from {username='oryx', principal='oryx/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:53670
I20260618 23:24:47.678234 16410 raft_consensus.cc:2229] T 3d837daf717d4ccfb8b7845523edd58c P adc409b2a6554449a338c9c17ca51855 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:24:47.678270 16410 raft_consensus.cc:2258] T 3d837daf717d4ccfb8b7845523edd58c P adc409b2a6554449a338c9c17ca51855 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:47.678285 16408 tablet_replica.cc:330] T 3d837daf717d4ccfb8b7845523edd58c P b725d44cdbe148c2864a6b795ab02019: stopping tablet replica
I20260618 23:24:47.678330 16408 raft_consensus.cc:2229] T 3d837daf717d4ccfb8b7845523edd58c P b725d44cdbe148c2864a6b795ab02019 [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:24:47.678402 16408 raft_consensus.cc:2258] T 3d837daf717d4ccfb8b7845523edd58c P b725d44cdbe148c2864a6b795ab02019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:47.678629 16410 ts_tablet_manager.cc:1905] T 3d837daf717d4ccfb8b7845523edd58c P adc409b2a6554449a338c9c17ca51855: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:47.678785 16408 ts_tablet_manager.cc:1905] T 3d837daf717d4ccfb8b7845523edd58c P b725d44cdbe148c2864a6b795ab02019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:47.680322 16408 ts_tablet_manager.cc:1918] T 3d837daf717d4ccfb8b7845523edd58c P b725d44cdbe148c2864a6b795ab02019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 23:24:47.680380 16408 log.cc:1196] T 3d837daf717d4ccfb8b7845523edd58c P b725d44cdbe148c2864a6b795ab02019: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-1/wal/wals/3d837daf717d4ccfb8b7845523edd58c
I20260618 23:24:47.680583 16408 ts_tablet_manager.cc:1939] T 3d837daf717d4ccfb8b7845523edd58c P b725d44cdbe148c2864a6b795ab02019: Deleting consensus metadata
I20260618 23:24:47.681707 16410 ts_tablet_manager.cc:1918] T 3d837daf717d4ccfb8b7845523edd58c P adc409b2a6554449a338c9c17ca51855: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 23:24:47.681761 16410 log.cc:1196] T 3d837daf717d4ccfb8b7845523edd58c P adc409b2a6554449a338c9c17ca51855: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-2/wal/wals/3d837daf717d4ccfb8b7845523edd58c
I20260618 23:24:47.681967 16410 ts_tablet_manager.cc:1939] T 3d837daf717d4ccfb8b7845523edd58c P adc409b2a6554449a338c9c17ca51855: Deleting consensus metadata
I20260618 23:24:47.682278 15859 catalog_manager.cc:4841] TS adc409b2a6554449a338c9c17ca51855 (127.8.39.131:43301): tablet 3d837daf717d4ccfb8b7845523edd58c (table test-table [id=4aadf7c6a4774856a746ca45c1dfa5f0]) successfully deleted
I20260618 23:24:47.682608 16012 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d837daf717d4ccfb8b7845523edd58c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:47 UTC) from {username='oryx', principal='oryx/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:41860
I20260618 23:24:47.682664 15861 catalog_manager.cc:4841] TS b725d44cdbe148c2864a6b795ab02019 (127.8.39.130:38975): tablet 3d837daf717d4ccfb8b7845523edd58c (table test-table [id=4aadf7c6a4774856a746ca45c1dfa5f0]) successfully deleted
I20260618 23:24:47.682710 16409 tablet_replica.cc:330] T 3d837daf717d4ccfb8b7845523edd58c P 3cffed2d98bf4816b9d3d4a263e62b95: stopping tablet replica
I20260618 23:24:47.682754 16409 raft_consensus.cc:2229] T 3d837daf717d4ccfb8b7845523edd58c P 3cffed2d98bf4816b9d3d4a263e62b95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:24:47.682801 16409 raft_consensus.cc:2258] T 3d837daf717d4ccfb8b7845523edd58c P 3cffed2d98bf4816b9d3d4a263e62b95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:47.683053 16409 ts_tablet_manager.cc:1905] T 3d837daf717d4ccfb8b7845523edd58c P 3cffed2d98bf4816b9d3d4a263e62b95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:47.684254 16409 ts_tablet_manager.cc:1918] T 3d837daf717d4ccfb8b7845523edd58c P 3cffed2d98bf4816b9d3d4a263e62b95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 23:24:47.684314 16409 log.cc:1196] T 3d837daf717d4ccfb8b7845523edd58c P 3cffed2d98bf4816b9d3d4a263e62b95: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-0/wal/wals/3d837daf717d4ccfb8b7845523edd58c
I20260618 23:24:47.684511 16409 ts_tablet_manager.cc:1939] T 3d837daf717d4ccfb8b7845523edd58c P 3cffed2d98bf4816b9d3d4a263e62b95: Deleting consensus metadata
I20260618 23:24:47.684808 15860 catalog_manager.cc:4841] TS 3cffed2d98bf4816b9d3d4a263e62b95 (127.8.39.129:40645): tablet 3d837daf717d4ccfb8b7845523edd58c (table test-table [id=4aadf7c6a4774856a746ca45c1dfa5f0]) successfully deleted
Jun 18 23:24:47 dist-test-slave-47qt krb5kdc[15830](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 18 23:24:47 dist-test-slave-47qt krb5kdc[15830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:47 dist-test-slave-47qt krb5kdc[15830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
W20260618 23:24:47.705021 15924 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:47.698740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38050 (local address 127.8.39.190:44111)
0618 23:24:47.698808 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:47.698810 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:47.698940 (+ 130us) server_negotiation.cc:407] Connection header received
0618 23:24:47.699020 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:47.699021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:47.699030 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:47.699071 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:47.699806 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.700303 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.701438 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.701623 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.701689 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:47.704954 (+ 3265us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38050: BlockingRecv error: recv got EOF from 127.0.0.1:38050 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Jun 18 23:24:47 dist-test-slave-47qt krb5kdc[15830](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825087, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM
I20260618 23:24:47.713526 15924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:47.706123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38064 (local address 127.8.39.190:44111)
0618 23:24:47.706161 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:47.706162 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:47.706212 (+ 50us) server_negotiation.cc:407] Connection header received
0618 23:24:47.706301 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:47.706302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:47.706309 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:47.706356 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:47.706934 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.707419 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.708392 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:47.708565 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:47.708621 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:47.710929 (+ 2308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:47.710931 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:47.710931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:47.710938 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:47.712394 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:47.712707 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:47.712708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:47.712709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:47.712767 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:47.712957 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:47.712958 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:47.712959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:47.713083 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:47.713230 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:47.713362 (+ 132us) server_negotiation.cc:299] Negotiation successful
0618 23:24:47.713435 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260618 23:24:47.715216 15875 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:24:47.715583 15875 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:24:47.717027 16012 tablet_service.cc:1467] Processing CreateTablet for tablet c2ad77226489443d83d18b9e9edb2a3c (DEFAULT_TABLE table=test-table [id=5296e8ec25cb4eb698cc08d89aec5042]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:47.717024 16278 tablet_service.cc:1467] Processing CreateTablet for tablet c2ad77226489443d83d18b9e9edb2a3c (DEFAULT_TABLE table=test-table [id=5296e8ec25cb4eb698cc08d89aec5042]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:47.717037 16152 tablet_service.cc:1467] Processing CreateTablet for tablet c2ad77226489443d83d18b9e9edb2a3c (DEFAULT_TABLE table=test-table [id=5296e8ec25cb4eb698cc08d89aec5042]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:47.717177 16278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2ad77226489443d83d18b9e9edb2a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:47.717177 16012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2ad77226489443d83d18b9e9edb2a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:47.717188 16152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2ad77226489443d83d18b9e9edb2a3c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:47.718467 16388 tablet_bootstrap.cc:492] T c2ad77226489443d83d18b9e9edb2a3c P b725d44cdbe148c2864a6b795ab02019: Bootstrap starting.
I20260618 23:24:47.718636 16387 tablet_bootstrap.cc:492] T c2ad77226489443d83d18b9e9edb2a3c P adc409b2a6554449a338c9c17ca51855: Bootstrap starting.
I20260618 23:24:47.718881 16388 tablet_bootstrap.cc:654] T c2ad77226489443d83d18b9e9edb2a3c P b725d44cdbe148c2864a6b795ab02019: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:47.719007 16387 tablet_bootstrap.cc:654] T c2ad77226489443d83d18b9e9edb2a3c P adc409b2a6554449a338c9c17ca51855: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:47.719221 16386 tablet_bootstrap.cc:492] T c2ad77226489443d83d18b9e9edb2a3c P 3cffed2d98bf4816b9d3d4a263e62b95: Bootstrap starting.
I20260618 23:24:47.719550 16386 tablet_bootstrap.cc:654] T c2ad77226489443d83d18b9e9edb2a3c P 3cffed2d98bf4816b9d3d4a263e62b95: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:47.719647 16387 tablet_bootstrap.cc:492] T c2ad77226489443d83d18b9e9edb2a3c P adc409b2a6554449a338c9c17ca51855: No bootstrap required, opened a new log
I20260618 23:24:47.719698 16387 ts_tablet_manager.cc:1397] T c2ad77226489443d83d18b9e9edb2a3c P adc409b2a6554449a338c9c17ca51855: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260618 23:24:47.719851 16387 raft_consensus.cc:348] T c2ad77226489443d83d18b9e9edb2a3c P adc409b2a6554449a338c9c17ca51855 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:47.719925 16387 raft_consensus.cc:374] T c2ad77226489443d83d18b9e9edb2a3c P adc409b2a6554449a338c9c17ca51855 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:47.719949 16387 raft_consensus.cc:729] T c2ad77226489443d83d18b9e9edb2a3c P adc409b2a6554449a338c9c17ca51855 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adc409b2a6554449a338c9c17ca51855, State: Initialized, Role: FOLLOWER
I20260618 23:24:47.719995 16387 consensus_queue.cc:260] T c2ad77226489443d83d18b9e9edb2a3c P adc409b2a6554449a338c9c17ca51855 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:47.720079 16387 ts_tablet_manager.cc:1428] T c2ad77226489443d83d18b9e9edb2a3c P adc409b2a6554449a338c9c17ca51855: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:47.720391 16388 tablet_bootstrap.cc:492] T c2ad77226489443d83d18b9e9edb2a3c P b725d44cdbe148c2864a6b795ab02019: No bootstrap required, opened a new log
I20260618 23:24:47.720446 16388 ts_tablet_manager.cc:1397] T c2ad77226489443d83d18b9e9edb2a3c P b725d44cdbe148c2864a6b795ab02019: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:24:47.720587 16388 raft_consensus.cc:348] T c2ad77226489443d83d18b9e9edb2a3c P b725d44cdbe148c2864a6b795ab02019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:47.720631 16388 raft_consensus.cc:374] T c2ad77226489443d83d18b9e9edb2a3c P b725d44cdbe148c2864a6b795ab02019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:47.720645 16388 raft_consensus.cc:729] T c2ad77226489443d83d18b9e9edb2a3c P b725d44cdbe148c2864a6b795ab02019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b725d44cdbe148c2864a6b795ab02019, State: Initialized, Role: FOLLOWER
I20260618 23:24:47.720638 16386 tablet_bootstrap.cc:492] T c2ad77226489443d83d18b9e9edb2a3c P 3cffed2d98bf4816b9d3d4a263e62b95: No bootstrap required, opened a new log
I20260618 23:24:47.720680 16386 ts_tablet_manager.cc:1397] T c2ad77226489443d83d18b9e9edb2a3c P 3cffed2d98bf4816b9d3d4a263e62b95: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260618 23:24:47.720702 16388 consensus_queue.cc:260] T c2ad77226489443d83d18b9e9edb2a3c P b725d44cdbe148c2864a6b795ab02019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:47.720798 16388 ts_tablet_manager.cc:1428] T c2ad77226489443d83d18b9e9edb2a3c P b725d44cdbe148c2864a6b795ab02019: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:47.720777 16386 raft_consensus.cc:348] T c2ad77226489443d83d18b9e9edb2a3c P 3cffed2d98bf4816b9d3d4a263e62b95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:47.720840 16386 raft_consensus.cc:374] T c2ad77226489443d83d18b9e9edb2a3c P 3cffed2d98bf4816b9d3d4a263e62b95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:47.721052 16386 raft_consensus.cc:729] T c2ad77226489443d83d18b9e9edb2a3c P 3cffed2d98bf4816b9d3d4a263e62b95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cffed2d98bf4816b9d3d4a263e62b95, State: Initialized, Role: FOLLOWER
I20260618 23:24:47.721312 16386 consensus_queue.cc:260] T c2ad77226489443d83d18b9e9edb2a3c P 3cffed2d98bf4816b9d3d4a263e62b95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:47.721424 16386 ts_tablet_manager.cc:1428] T c2ad77226489443d83d18b9e9edb2a3c P 3cffed2d98bf4816b9d3d4a263e62b95: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260618 23:24:48.051095 16393 raft_consensus.cc:482] T c2ad77226489443d83d18b9e9edb2a3c P adc409b2a6554449a338c9c17ca51855 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:24:48.051213 16393 raft_consensus.cc:504] T c2ad77226489443d83d18b9e9edb2a3c P adc409b2a6554449a338c9c17ca51855 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:48.051585 16393 leader_election.cc:288] T c2ad77226489443d83d18b9e9edb2a3c P adc409b2a6554449a338c9c17ca51855 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cffed2d98bf4816b9d3d4a263e62b95 (127.8.39.129:40645), b725d44cdbe148c2864a6b795ab02019 (127.8.39.130:38975)
I20260618 23:24:48.055817 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:48.051934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:36297 (local address 127.8.39.130:38975)
0618 23:24:48.051981 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:48.051984 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:48.052026 (+ 42us) server_negotiation.cc:407] Connection header received
0618 23:24:48.052092 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:48.052093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:48.052104 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:48.052174 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:48.052453 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:48.053253 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:48.054829 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:48.055363 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:48.055444 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:48.055455 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:48.055707 (+ 252us) server_negotiation.cc:299] Negotiation successful
0618 23:24:48.055749 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260618 23:24:48.055835 16457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:48.051880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:48311 (local address 127.8.39.129:40645)
0618 23:24:48.052009 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:48.052011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:48.052021 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:24:48.052112 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:48.052113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:48.052124 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:48.052184 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:24:48.052454 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:48.053259 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:48.054824 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:48.055418 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:48.055484 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:48.055492 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:48.055707 (+ 215us) server_negotiation.cc:299] Negotiation successful
0618 23:24:48.055752 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":56,"threads_started":1}
I20260618 23:24:48.055845 16456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:48.051807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:40645 (local address 127.8.39.131:48311)
0618 23:24:48.051939 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:48.051952 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:48.052022 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:48.052237 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:48.052238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:48.052242 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:48.052350 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:48.052352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:48.053392 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:48.053392 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:48.054713 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:48.054715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:48.055521 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:48.055522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:48.055578 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:48.055581 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:24:48.055645 (+ 64us) client_negotiation.cc:240] Negotiation successful
0618 23:24:48.055706 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":94,"thread_start_us":69,"threads_started":1}
I20260618 23:24:48.056152 16455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:48.051813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:38975 (local address 127.8.39.131:36297)
0618 23:24:48.051939 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:48.051952 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:48.052032 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:48.052237 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:48.052238 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:48.052240 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:24:48.052331 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:48.052333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:48.053374 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:48.053375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:48.054713 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:48.054715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:48.055485 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:48.055486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:48.055567 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:48.055573 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 23:24:48.055645 (+ 72us) client_negotiation.cc:240] Negotiation successful
0618 23:24:48.055706 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":63,"spinlock_wait_cycles":896,"thread_start_us":65,"threads_started":1}
I20260618 23:24:48.056139 16032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2ad77226489443d83d18b9e9edb2a3c" candidate_uuid: "adc409b2a6554449a338c9c17ca51855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" is_pre_election: true
I20260618 23:24:48.056234 16032 raft_consensus.cc:2454] T c2ad77226489443d83d18b9e9edb2a3c P 3cffed2d98bf4816b9d3d4a263e62b95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adc409b2a6554449a338c9c17ca51855 in term 0.
I20260618 23:24:48.056399 16244 leader_election.cc:302] T c2ad77226489443d83d18b9e9edb2a3c P adc409b2a6554449a338c9c17ca51855 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cffed2d98bf4816b9d3d4a263e62b95, adc409b2a6554449a338c9c17ca51855; no voters:
I20260618 23:24:48.056429 16172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2ad77226489443d83d18b9e9edb2a3c" candidate_uuid: "adc409b2a6554449a338c9c17ca51855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b725d44cdbe148c2864a6b795ab02019" is_pre_election: true
I20260618 23:24:48.056519 16172 raft_consensus.cc:2454] T c2ad77226489443d83d18b9e9edb2a3c P b725d44cdbe148c2864a6b795ab02019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adc409b2a6554449a338c9c17ca51855 in term 0.
I20260618 23:24:48.056569 16393 raft_consensus.cc:2781] T c2ad77226489443d83d18b9e9edb2a3c P adc409b2a6554449a338c9c17ca51855 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:24:48.056608 16393 raft_consensus.cc:482] T c2ad77226489443d83d18b9e9edb2a3c P adc409b2a6554449a338c9c17ca51855 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:24:48.056635 16393 raft_consensus.cc:3037] T c2ad77226489443d83d18b9e9edb2a3c P adc409b2a6554449a338c9c17ca51855 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:48.057354 16393 raft_consensus.cc:504] T c2ad77226489443d83d18b9e9edb2a3c P adc409b2a6554449a338c9c17ca51855 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:48.057498 16393 leader_election.cc:288] T c2ad77226489443d83d18b9e9edb2a3c P adc409b2a6554449a338c9c17ca51855 [CANDIDATE]: Term 1 election: Requested vote from peers 3cffed2d98bf4816b9d3d4a263e62b95 (127.8.39.129:40645), b725d44cdbe148c2864a6b795ab02019 (127.8.39.130:38975)
I20260618 23:24:48.057631 16032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2ad77226489443d83d18b9e9edb2a3c" candidate_uuid: "adc409b2a6554449a338c9c17ca51855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cffed2d98bf4816b9d3d4a263e62b95"
I20260618 23:24:48.057700 16032 raft_consensus.cc:3037] T c2ad77226489443d83d18b9e9edb2a3c P 3cffed2d98bf4816b9d3d4a263e62b95 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:48.057711 16172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2ad77226489443d83d18b9e9edb2a3c" candidate_uuid: "adc409b2a6554449a338c9c17ca51855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b725d44cdbe148c2864a6b795ab02019"
I20260618 23:24:48.057790 16172 raft_consensus.cc:3037] T c2ad77226489443d83d18b9e9edb2a3c P b725d44cdbe148c2864a6b795ab02019 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:48.058323 16172 raft_consensus.cc:2454] T c2ad77226489443d83d18b9e9edb2a3c P b725d44cdbe148c2864a6b795ab02019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adc409b2a6554449a338c9c17ca51855 in term 1.
I20260618 23:24:48.058359 16032 raft_consensus.cc:2454] T c2ad77226489443d83d18b9e9edb2a3c P 3cffed2d98bf4816b9d3d4a263e62b95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adc409b2a6554449a338c9c17ca51855 in term 1.
I20260618 23:24:48.058490 16245 leader_election.cc:302] T c2ad77226489443d83d18b9e9edb2a3c P adc409b2a6554449a338c9c17ca51855 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adc409b2a6554449a338c9c17ca51855, b725d44cdbe148c2864a6b795ab02019; no voters:
I20260618 23:24:48.058588 16393 raft_consensus.cc:2781] T c2ad77226489443d83d18b9e9edb2a3c P adc409b2a6554449a338c9c17ca51855 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:48.058706 16393 raft_consensus.cc:686] T c2ad77226489443d83d18b9e9edb2a3c P adc409b2a6554449a338c9c17ca51855 [term 1 LEADER]: Becoming Leader. State: Replica: adc409b2a6554449a338c9c17ca51855, State: Running, Role: LEADER
I20260618 23:24:48.058779 16393 consensus_queue.cc:237] T c2ad77226489443d83d18b9e9edb2a3c P adc409b2a6554449a338c9c17ca51855 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:48.059427 15875 catalog_manager.cc:5495] T c2ad77226489443d83d18b9e9edb2a3c P adc409b2a6554449a338c9c17ca51855 reported cstate change: term changed from 0 to 1, leader changed from <none> to adc409b2a6554449a338c9c17ca51855 (127.8.39.131). New cstate: current_term: 1 leader_uuid: "adc409b2a6554449a338c9c17ca51855" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } health_report { overall_health: HEALTHY } } }
I20260618 23:24:48.076186 16461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:48.073709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47054 (local address 127.8.39.131:43301)
0618 23:24:48.073840 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:48.073841 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:48.073847 (+ 6us) server_negotiation.cc:407] Connection header received
0618 23:24:48.073876 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:48.073876 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:48.073892 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:48.073988 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:24:48.074135 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:48.074619 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:48.075387 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:48.075580 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:48.075664 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:48.075853 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:24:48.075956 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:24:48.076040 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:48.076080 (+ 40us) server_negotiation.cc:299] Negotiation successful
0618 23:24:48.076099 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":91,"threads_started":1}
I20260618 23:24:48.077277 16032 raft_consensus.cc:1261] T c2ad77226489443d83d18b9e9edb2a3c P 3cffed2d98bf4816b9d3d4a263e62b95 [term 1 FOLLOWER]: Refusing update from remote peer adc409b2a6554449a338c9c17ca51855: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:24:48.077286 16172 raft_consensus.cc:1261] T c2ad77226489443d83d18b9e9edb2a3c P b725d44cdbe148c2864a6b795ab02019 [term 1 FOLLOWER]: Refusing update from remote peer adc409b2a6554449a338c9c17ca51855: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:24:48.077643 16393 consensus_queue.cc:1035] T c2ad77226489443d83d18b9e9edb2a3c P adc409b2a6554449a338c9c17ca51855 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:48.077867 16458 consensus_queue.cc:1035] T c2ad77226489443d83d18b9e9edb2a3c P adc409b2a6554449a338c9c17ca51855 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:48.081559 15875 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:24:48.081638 15875 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:24:48.082239 15875 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71871ac5b89340e7bd6204518a23b6b8: Sending DeleteTablet for 3 replicas of tablet c2ad77226489443d83d18b9e9edb2a3c
I20260618 23:24:48.082479 16012 tablet_service.cc:1514] Processing DeleteTablet for tablet c2ad77226489443d83d18b9e9edb2a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:48 UTC) from {username='oryx', principal='oryx/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:41860
I20260618 23:24:48.082504 16278 tablet_service.cc:1514] Processing DeleteTablet for tablet c2ad77226489443d83d18b9e9edb2a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:48 UTC) from {username='oryx', principal='oryx/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:47048
I20260618 23:24:48.082585 16410 tablet_replica.cc:330] T c2ad77226489443d83d18b9e9edb2a3c P adc409b2a6554449a338c9c17ca51855: stopping tablet replica
I20260618 23:24:48.082576 16152 tablet_service.cc:1514] Processing DeleteTablet for tablet c2ad77226489443d83d18b9e9edb2a3c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:48 UTC) from {username='oryx', principal='oryx/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:53670
I20260618 23:24:48.082631 16410 raft_consensus.cc:2229] T c2ad77226489443d83d18b9e9edb2a3c P adc409b2a6554449a338c9c17ca51855 [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:24:48.082655 16409 tablet_replica.cc:330] T c2ad77226489443d83d18b9e9edb2a3c P 3cffed2d98bf4816b9d3d4a263e62b95: stopping tablet replica
I20260618 23:24:48.082705 16410 raft_consensus.cc:2258] T c2ad77226489443d83d18b9e9edb2a3c P adc409b2a6554449a338c9c17ca51855 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:48.082710 16409 raft_consensus.cc:2229] T c2ad77226489443d83d18b9e9edb2a3c P 3cffed2d98bf4816b9d3d4a263e62b95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:24:48.082754 16409 raft_consensus.cc:2258] T c2ad77226489443d83d18b9e9edb2a3c P 3cffed2d98bf4816b9d3d4a263e62b95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:48.082821 16408 tablet_replica.cc:330] T c2ad77226489443d83d18b9e9edb2a3c P b725d44cdbe148c2864a6b795ab02019: stopping tablet replica
I20260618 23:24:48.082764 15875 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260618 23:24:48.082870 16408 raft_consensus.cc:2229] T c2ad77226489443d83d18b9e9edb2a3c P b725d44cdbe148c2864a6b795ab02019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:24:48.082911 16408 raft_consensus.cc:2258] T c2ad77226489443d83d18b9e9edb2a3c P b725d44cdbe148c2864a6b795ab02019 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260618 23:24:48.082911 15875 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:24:48.083012 16410 ts_tablet_manager.cc:1905] T c2ad77226489443d83d18b9e9edb2a3c P adc409b2a6554449a338c9c17ca51855: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:48.083375 16408 ts_tablet_manager.cc:1905] T c2ad77226489443d83d18b9e9edb2a3c P b725d44cdbe148c2864a6b795ab02019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:48.083407 16409 ts_tablet_manager.cc:1905] T c2ad77226489443d83d18b9e9edb2a3c P 3cffed2d98bf4816b9d3d4a263e62b95: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:48.084638 16410 ts_tablet_manager.cc:1918] T c2ad77226489443d83d18b9e9edb2a3c P adc409b2a6554449a338c9c17ca51855: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:24:48.084692 16410 log.cc:1196] T c2ad77226489443d83d18b9e9edb2a3c P adc409b2a6554449a338c9c17ca51855: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-2/wal/wals/c2ad77226489443d83d18b9e9edb2a3c
I20260618 23:24:48.084818 16408 ts_tablet_manager.cc:1918] T c2ad77226489443d83d18b9e9edb2a3c P b725d44cdbe148c2864a6b795ab02019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:24:48.084847 16409 ts_tablet_manager.cc:1918] T c2ad77226489443d83d18b9e9edb2a3c P 3cffed2d98bf4816b9d3d4a263e62b95: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:24:48.084872 16408 log.cc:1196] T c2ad77226489443d83d18b9e9edb2a3c P b725d44cdbe148c2864a6b795ab02019: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-1/wal/wals/c2ad77226489443d83d18b9e9edb2a3c
I20260618 23:24:48.084893 16409 log.cc:1196] T c2ad77226489443d83d18b9e9edb2a3c P 3cffed2d98bf4816b9d3d4a263e62b95: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-0/wal/wals/c2ad77226489443d83d18b9e9edb2a3c
I20260618 23:24:48.084908 16410 ts_tablet_manager.cc:1939] T c2ad77226489443d83d18b9e9edb2a3c P adc409b2a6554449a338c9c17ca51855: Deleting consensus metadata
I20260618 23:24:48.085090 16409 ts_tablet_manager.cc:1939] T c2ad77226489443d83d18b9e9edb2a3c P 3cffed2d98bf4816b9d3d4a263e62b95: Deleting consensus metadata
I20260618 23:24:48.085090 16408 ts_tablet_manager.cc:1939] T c2ad77226489443d83d18b9e9edb2a3c P b725d44cdbe148c2864a6b795ab02019: Deleting consensus metadata
I20260618 23:24:48.085438 15861 catalog_manager.cc:4841] TS b725d44cdbe148c2864a6b795ab02019 (127.8.39.130:38975): tablet c2ad77226489443d83d18b9e9edb2a3c (table test-table [id=5296e8ec25cb4eb698cc08d89aec5042]) successfully deleted
I20260618 23:24:48.085502 15860 catalog_manager.cc:4841] TS 3cffed2d98bf4816b9d3d4a263e62b95 (127.8.39.129:40645): tablet c2ad77226489443d83d18b9e9edb2a3c (table test-table [id=5296e8ec25cb4eb698cc08d89aec5042]) successfully deleted
I20260618 23:24:48.086282 15859 catalog_manager.cc:4841] TS adc409b2a6554449a338c9c17ca51855 (127.8.39.131:43301): tablet c2ad77226489443d83d18b9e9edb2a3c (table test-table [id=5296e8ec25cb4eb698cc08d89aec5042]) successfully deleted
I20260618 23:24:48.086926 16152 tablet_service.cc:1467] Processing CreateTablet for tablet 87666ad7fd0b49048d4b3e059a4370dd (DEFAULT_TABLE table=test-table [id=3ab8301b6bff41678c74f36b5e55e93e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:48.086982 16012 tablet_service.cc:1467] Processing CreateTablet for tablet 87666ad7fd0b49048d4b3e059a4370dd (DEFAULT_TABLE table=test-table [id=3ab8301b6bff41678c74f36b5e55e93e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:48.087034 16152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87666ad7fd0b49048d4b3e059a4370dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:48.087033 16278 tablet_service.cc:1467] Processing CreateTablet for tablet 87666ad7fd0b49048d4b3e059a4370dd (DEFAULT_TABLE table=test-table [id=3ab8301b6bff41678c74f36b5e55e93e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:24:48.087067 16012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87666ad7fd0b49048d4b3e059a4370dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:48.087132 16278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87666ad7fd0b49048d4b3e059a4370dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:48.088161 16386 tablet_bootstrap.cc:492] T 87666ad7fd0b49048d4b3e059a4370dd P 3cffed2d98bf4816b9d3d4a263e62b95: Bootstrap starting.
I20260618 23:24:48.088289 16387 tablet_bootstrap.cc:492] T 87666ad7fd0b49048d4b3e059a4370dd P adc409b2a6554449a338c9c17ca51855: Bootstrap starting.
I20260618 23:24:48.088301 16388 tablet_bootstrap.cc:492] T 87666ad7fd0b49048d4b3e059a4370dd P b725d44cdbe148c2864a6b795ab02019: Bootstrap starting.
I20260618 23:24:48.088469 16386 tablet_bootstrap.cc:654] T 87666ad7fd0b49048d4b3e059a4370dd P 3cffed2d98bf4816b9d3d4a263e62b95: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:48.088613 16388 tablet_bootstrap.cc:654] T 87666ad7fd0b49048d4b3e059a4370dd P b725d44cdbe148c2864a6b795ab02019: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:48.088681 16387 tablet_bootstrap.cc:654] T 87666ad7fd0b49048d4b3e059a4370dd P adc409b2a6554449a338c9c17ca51855: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:48.089097 16386 tablet_bootstrap.cc:492] T 87666ad7fd0b49048d4b3e059a4370dd P 3cffed2d98bf4816b9d3d4a263e62b95: No bootstrap required, opened a new log
I20260618 23:24:48.089178 16386 ts_tablet_manager.cc:1397] T 87666ad7fd0b49048d4b3e059a4370dd P 3cffed2d98bf4816b9d3d4a263e62b95: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260618 23:24:48.089293 16386 raft_consensus.cc:348] T 87666ad7fd0b49048d4b3e059a4370dd P 3cffed2d98bf4816b9d3d4a263e62b95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:48.089370 16386 raft_consensus.cc:374] T 87666ad7fd0b49048d4b3e059a4370dd P 3cffed2d98bf4816b9d3d4a263e62b95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:48.089395 16386 raft_consensus.cc:729] T 87666ad7fd0b49048d4b3e059a4370dd P 3cffed2d98bf4816b9d3d4a263e62b95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cffed2d98bf4816b9d3d4a263e62b95, State: Initialized, Role: FOLLOWER
I20260618 23:24:48.089443 16386 consensus_queue.cc:260] T 87666ad7fd0b49048d4b3e059a4370dd P 3cffed2d98bf4816b9d3d4a263e62b95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:48.089519 16387 tablet_bootstrap.cc:492] T 87666ad7fd0b49048d4b3e059a4370dd P adc409b2a6554449a338c9c17ca51855: No bootstrap required, opened a new log
I20260618 23:24:48.089540 16386 ts_tablet_manager.cc:1428] T 87666ad7fd0b49048d4b3e059a4370dd P 3cffed2d98bf4816b9d3d4a263e62b95: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:48.089567 16387 ts_tablet_manager.cc:1397] T 87666ad7fd0b49048d4b3e059a4370dd P adc409b2a6554449a338c9c17ca51855: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260618 23:24:48.089710 16387 raft_consensus.cc:348] T 87666ad7fd0b49048d4b3e059a4370dd P adc409b2a6554449a338c9c17ca51855 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:48.089764 16387 raft_consensus.cc:374] T 87666ad7fd0b49048d4b3e059a4370dd P adc409b2a6554449a338c9c17ca51855 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:48.089777 16387 raft_consensus.cc:729] T 87666ad7fd0b49048d4b3e059a4370dd P adc409b2a6554449a338c9c17ca51855 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adc409b2a6554449a338c9c17ca51855, State: Initialized, Role: FOLLOWER
I20260618 23:24:48.089836 16387 consensus_queue.cc:260] T 87666ad7fd0b49048d4b3e059a4370dd P adc409b2a6554449a338c9c17ca51855 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:48.089921 16387 ts_tablet_manager.cc:1428] T 87666ad7fd0b49048d4b3e059a4370dd P adc409b2a6554449a338c9c17ca51855: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:48.090762 16388 tablet_bootstrap.cc:492] T 87666ad7fd0b49048d4b3e059a4370dd P b725d44cdbe148c2864a6b795ab02019: No bootstrap required, opened a new log
I20260618 23:24:48.090817 16388 ts_tablet_manager.cc:1397] T 87666ad7fd0b49048d4b3e059a4370dd P b725d44cdbe148c2864a6b795ab02019: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260618 23:24:48.090925 16388 raft_consensus.cc:348] T 87666ad7fd0b49048d4b3e059a4370dd P b725d44cdbe148c2864a6b795ab02019 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:48.090993 16388 raft_consensus.cc:374] T 87666ad7fd0b49048d4b3e059a4370dd P b725d44cdbe148c2864a6b795ab02019 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:48.091022 16388 raft_consensus.cc:729] T 87666ad7fd0b49048d4b3e059a4370dd P b725d44cdbe148c2864a6b795ab02019 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b725d44cdbe148c2864a6b795ab02019, State: Initialized, Role: FOLLOWER
I20260618 23:24:48.091061 16388 consensus_queue.cc:260] T 87666ad7fd0b49048d4b3e059a4370dd P b725d44cdbe148c2864a6b795ab02019 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:48.091152 16388 ts_tablet_manager.cc:1428] T 87666ad7fd0b49048d4b3e059a4370dd P b725d44cdbe148c2864a6b795ab02019: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:48.119240 16464 raft_consensus.cc:482] T 87666ad7fd0b49048d4b3e059a4370dd P adc409b2a6554449a338c9c17ca51855 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:24:48.119360 16464 raft_consensus.cc:504] T 87666ad7fd0b49048d4b3e059a4370dd P adc409b2a6554449a338c9c17ca51855 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:48.119531 16464 leader_election.cc:288] T 87666ad7fd0b49048d4b3e059a4370dd P adc409b2a6554449a338c9c17ca51855 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cffed2d98bf4816b9d3d4a263e62b95 (127.8.39.129:40645), b725d44cdbe148c2864a6b795ab02019 (127.8.39.130:38975)
I20260618 23:24:48.119776 16032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87666ad7fd0b49048d4b3e059a4370dd" candidate_uuid: "adc409b2a6554449a338c9c17ca51855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" is_pre_election: true
I20260618 23:24:48.119846 16172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87666ad7fd0b49048d4b3e059a4370dd" candidate_uuid: "adc409b2a6554449a338c9c17ca51855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b725d44cdbe148c2864a6b795ab02019" is_pre_election: true
I20260618 23:24:48.119910 16032 raft_consensus.cc:2454] T 87666ad7fd0b49048d4b3e059a4370dd P 3cffed2d98bf4816b9d3d4a263e62b95 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adc409b2a6554449a338c9c17ca51855 in term 0.
I20260618 23:24:48.119933 16172 raft_consensus.cc:2454] T 87666ad7fd0b49048d4b3e059a4370dd P b725d44cdbe148c2864a6b795ab02019 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate adc409b2a6554449a338c9c17ca51855 in term 0.
I20260618 23:24:48.120057 16244 leader_election.cc:302] T 87666ad7fd0b49048d4b3e059a4370dd P adc409b2a6554449a338c9c17ca51855 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cffed2d98bf4816b9d3d4a263e62b95, adc409b2a6554449a338c9c17ca51855; no voters:
I20260618 23:24:48.120147 16464 raft_consensus.cc:2781] T 87666ad7fd0b49048d4b3e059a4370dd P adc409b2a6554449a338c9c17ca51855 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:24:48.120196 16464 raft_consensus.cc:482] T 87666ad7fd0b49048d4b3e059a4370dd P adc409b2a6554449a338c9c17ca51855 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:24:48.120227 16464 raft_consensus.cc:3037] T 87666ad7fd0b49048d4b3e059a4370dd P adc409b2a6554449a338c9c17ca51855 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:48.121271 16464 raft_consensus.cc:504] T 87666ad7fd0b49048d4b3e059a4370dd P adc409b2a6554449a338c9c17ca51855 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:48.121423 16464 leader_election.cc:288] T 87666ad7fd0b49048d4b3e059a4370dd P adc409b2a6554449a338c9c17ca51855 [CANDIDATE]: Term 1 election: Requested vote from peers 3cffed2d98bf4816b9d3d4a263e62b95 (127.8.39.129:40645), b725d44cdbe148c2864a6b795ab02019 (127.8.39.130:38975)
I20260618 23:24:48.121582 16032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87666ad7fd0b49048d4b3e059a4370dd" candidate_uuid: "adc409b2a6554449a338c9c17ca51855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cffed2d98bf4816b9d3d4a263e62b95"
I20260618 23:24:48.121663 16032 raft_consensus.cc:3037] T 87666ad7fd0b49048d4b3e059a4370dd P 3cffed2d98bf4816b9d3d4a263e62b95 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:48.121639 16172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87666ad7fd0b49048d4b3e059a4370dd" candidate_uuid: "adc409b2a6554449a338c9c17ca51855" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b725d44cdbe148c2864a6b795ab02019"
I20260618 23:24:48.121728 16172 raft_consensus.cc:3037] T 87666ad7fd0b49048d4b3e059a4370dd P b725d44cdbe148c2864a6b795ab02019 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:48.122283 16172 raft_consensus.cc:2454] T 87666ad7fd0b49048d4b3e059a4370dd P b725d44cdbe148c2864a6b795ab02019 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adc409b2a6554449a338c9c17ca51855 in term 1.
I20260618 23:24:48.122414 16032 raft_consensus.cc:2454] T 87666ad7fd0b49048d4b3e059a4370dd P 3cffed2d98bf4816b9d3d4a263e62b95 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate adc409b2a6554449a338c9c17ca51855 in term 1.
I20260618 23:24:48.122452 16245 leader_election.cc:302] T 87666ad7fd0b49048d4b3e059a4370dd P adc409b2a6554449a338c9c17ca51855 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: adc409b2a6554449a338c9c17ca51855, b725d44cdbe148c2864a6b795ab02019; no voters:
I20260618 23:24:48.122535 16464 raft_consensus.cc:2781] T 87666ad7fd0b49048d4b3e059a4370dd P adc409b2a6554449a338c9c17ca51855 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:48.122578 16464 raft_consensus.cc:686] T 87666ad7fd0b49048d4b3e059a4370dd P adc409b2a6554449a338c9c17ca51855 [term 1 LEADER]: Becoming Leader. State: Replica: adc409b2a6554449a338c9c17ca51855, State: Running, Role: LEADER
I20260618 23:24:48.122627 16464 consensus_queue.cc:237] T 87666ad7fd0b49048d4b3e059a4370dd P adc409b2a6554449a338c9c17ca51855 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } }
I20260618 23:24:48.123329 15874 catalog_manager.cc:5495] T 87666ad7fd0b49048d4b3e059a4370dd P adc409b2a6554449a338c9c17ca51855 reported cstate change: term changed from 0 to 1, leader changed from <none> to adc409b2a6554449a338c9c17ca51855 (127.8.39.131). New cstate: current_term: 1 leader_uuid: "adc409b2a6554449a338c9c17ca51855" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "adc409b2a6554449a338c9c17ca51855" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 43301 } health_report { overall_health: HEALTHY } } }
I20260618 23:24:48.132200 16383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:48.129600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:56215 (local address 127.8.39.130:38975)
0618 23:24:48.129642 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:48.129644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:48.129653 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:24:48.129717 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:48.129718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:48.129728 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:48.129797 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:24:48.129896 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:48.130639 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:48.131444 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:48.131716 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:48.131796 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:48.131953 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:24:48.131982 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:24:48.132032 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:48.132115 (+ 83us) server_negotiation.cc:299] Negotiation successful
0618 23:24:48.132134 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260618 23:24:48.132215 16472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:48.129466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:38975 (local address 127.8.39.131:56215)
0618 23:24:48.129603 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:48.129614 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:48.129661 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:48.129803 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:48.129804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:48.129805 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 23:24:48.129858 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:48.129859 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:48.130749 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:48.130750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:48.131360 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:48.131362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:48.131804 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:48.131805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:48.131857 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:48.131870 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 23:24:48.132061 (+ 191us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 23:24:48.132064 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:24:48.132111 (+ 47us) client_negotiation.cc:240] Negotiation successful
0618 23:24:48.132149 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":78,"threads_started":1}
I20260618 23:24:48.134872 16032 raft_consensus.cc:1261] T 87666ad7fd0b49048d4b3e059a4370dd P 3cffed2d98bf4816b9d3d4a263e62b95 [term 1 FOLLOWER]: Refusing update from remote peer adc409b2a6554449a338c9c17ca51855: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:24:48.134856 16172 raft_consensus.cc:1261] T 87666ad7fd0b49048d4b3e059a4370dd P b725d44cdbe148c2864a6b795ab02019 [term 1 FOLLOWER]: Refusing update from remote peer adc409b2a6554449a338c9c17ca51855: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:24:48.135056 16458 consensus_queue.cc:1035] T 87666ad7fd0b49048d4b3e059a4370dd P adc409b2a6554449a338c9c17ca51855 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cffed2d98bf4816b9d3d4a263e62b95" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40645 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:48.135207 16464 consensus_queue.cc:1035] T 87666ad7fd0b49048d4b3e059a4370dd P adc409b2a6554449a338c9c17ca51855 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b725d44cdbe148c2864a6b795ab02019" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 38975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:24:48.144601 16461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:48.141634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:48003 (local address 127.8.39.131:43301)
0618 23:24:48.141722 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:48.141724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:48.141732 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:24:48.141752 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:48.141753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:48.141764 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:48.141810 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:24:48.141976 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:48.142706 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:48.143834 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:48.144138 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:48.144205 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:48.144344 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:24:48.144379 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:24:48.144425 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:48.144504 (+ 79us) server_negotiation.cc:299] Negotiation successful
0618 23:24:48.144528 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260618 23:24:48.144619 16436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:48.141265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:43301 (local address 127.8.39.130:48003)
0618 23:24:48.141317 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:48.141329 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:48.141375 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:48.141855 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:48.141856 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:48.141857 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0618 23:24:48.141898 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:48.141899 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:48.142890 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:48.142891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:48.143748 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:48.143750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:48.144209 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:48.144210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:48.144267 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:48.144273 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0618 23:24:48.144451 (+ 178us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0618 23:24:48.144453 (+ 2us) client_negotiation.cc:769] Sending connection context
0618 23:24:48.144500 (+ 47us) client_negotiation.cc:240] Negotiation successful
0618 23:24:48.144546 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260618 23:24:48.151110 15875 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:24:48.151257 15875 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:24:48.151980 15875 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 71871ac5b89340e7bd6204518a23b6b8: Sending DeleteTablet for 3 replicas of tablet 87666ad7fd0b49048d4b3e059a4370dd
I20260618 23:24:48.152226 16278 tablet_service.cc:1514] Processing DeleteTablet for tablet 87666ad7fd0b49048d4b3e059a4370dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:48 UTC) from {username='oryx', principal='oryx/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:47048
I20260618 23:24:48.152315 16410 tablet_replica.cc:330] T 87666ad7fd0b49048d4b3e059a4370dd P adc409b2a6554449a338c9c17ca51855: stopping tablet replica
I20260618 23:24:48.152343 16012 tablet_service.cc:1514] Processing DeleteTablet for tablet 87666ad7fd0b49048d4b3e059a4370dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:48 UTC) from {username='oryx', principal='oryx/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:41860
I20260618 23:24:48.152372 16410 raft_consensus.cc:2229] T 87666ad7fd0b49048d4b3e059a4370dd P adc409b2a6554449a338c9c17ca51855 [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:24:48.152429 16410 raft_consensus.cc:2258] T 87666ad7fd0b49048d4b3e059a4370dd P adc409b2a6554449a338c9c17ca51855 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:48.152438 16409 tablet_replica.cc:330] T 87666ad7fd0b49048d4b3e059a4370dd P 3cffed2d98bf4816b9d3d4a263e62b95: stopping tablet replica
I20260618 23:24:48.152498 16409 raft_consensus.cc:2229] T 87666ad7fd0b49048d4b3e059a4370dd P 3cffed2d98bf4816b9d3d4a263e62b95 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:24:48.152539 16409 raft_consensus.cc:2258] T 87666ad7fd0b49048d4b3e059a4370dd P 3cffed2d98bf4816b9d3d4a263e62b95 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:48.152563 16152 tablet_service.cc:1514] Processing DeleteTablet for tablet 87666ad7fd0b49048d4b3e059a4370dd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:24:48 UTC) from {username='oryx', principal='oryx/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:53670
I20260618 23:24:48.152637 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 15947
I20260618 23:24:48.152730 16408 tablet_replica.cc:330] T 87666ad7fd0b49048d4b3e059a4370dd P b725d44cdbe148c2864a6b795ab02019: stopping tablet replica
I20260618 23:24:48.152782 16408 raft_consensus.cc:2229] T 87666ad7fd0b49048d4b3e059a4370dd P b725d44cdbe148c2864a6b795ab02019 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:24:48.152814 16408 raft_consensus.cc:2258] T 87666ad7fd0b49048d4b3e059a4370dd P b725d44cdbe148c2864a6b795ab02019 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:24:48.153519 16410 ts_tablet_manager.cc:1905] T 87666ad7fd0b49048d4b3e059a4370dd P adc409b2a6554449a338c9c17ca51855: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:48.154742 16408 ts_tablet_manager.cc:1905] T 87666ad7fd0b49048d4b3e059a4370dd P b725d44cdbe148c2864a6b795ab02019: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:24:48.154942 16410 ts_tablet_manager.cc:1918] T 87666ad7fd0b49048d4b3e059a4370dd P adc409b2a6554449a338c9c17ca51855: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 23:24:48.154994 16410 log.cc:1196] T 87666ad7fd0b49048d4b3e059a4370dd P adc409b2a6554449a338c9c17ca51855: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-2/wal/wals/87666ad7fd0b49048d4b3e059a4370dd
I20260618 23:24:48.155211 16410 ts_tablet_manager.cc:1939] T 87666ad7fd0b49048d4b3e059a4370dd P adc409b2a6554449a338c9c17ca51855: Deleting consensus metadata
I20260618 23:24:48.156244 15859 catalog_manager.cc:4841] TS adc409b2a6554449a338c9c17ca51855 (127.8.39.131:43301): tablet 87666ad7fd0b49048d4b3e059a4370dd (table test-table [id=3ab8301b6bff41678c74f36b5e55e93e]) successfully deleted
I20260618 23:24:48.156427 16408 ts_tablet_manager.cc:1918] T 87666ad7fd0b49048d4b3e059a4370dd P b725d44cdbe148c2864a6b795ab02019: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260618 23:24:48.156502 16408 log.cc:1196] T 87666ad7fd0b49048d4b3e059a4370dd P b725d44cdbe148c2864a6b795ab02019: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781825053542692-8350-0/minicluster-data/ts-1/wal/wals/87666ad7fd0b49048d4b3e059a4370dd
I20260618 23:24:48.156709 16408 ts_tablet_manager.cc:1939] T 87666ad7fd0b49048d4b3e059a4370dd P b725d44cdbe148c2864a6b795ab02019: Deleting consensus metadata
I20260618 23:24:48.157020 15861 catalog_manager.cc:4841] TS b725d44cdbe148c2864a6b795ab02019 (127.8.39.130:38975): tablet 87666ad7fd0b49048d4b3e059a4370dd (table test-table [id=3ab8301b6bff41678c74f36b5e55e93e]) successfully deleted
W20260618 23:24:48.161458 15860 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.39.129:40645 (error 108)
I20260618 23:24:48.162218 16476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:48.161864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:40645 (local address 127.0.0.1:41868)
0618 23:24:48.162078 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:48.162148 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.129:40645: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":44,"threads_started":1}
I20260618 23:24:48.162302 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 16087
W20260618 23:24:48.165002 15860 catalog_manager.cc:4584] TS 3cffed2d98bf4816b9d3d4a263e62b95 (127.8.39.129:40645): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 87666ad7fd0b49048d4b3e059a4370dd: Network error: Client connection negotiation failed: client connection to 127.8.39.129:40645: connect: Connection refused (error 111)
I20260618 23:24:48.171586 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 16227
I20260618 23:24:48.179297 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 15843
2026-06-18T23:24:48Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2931 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:24:48 dist-test-slave-47qt krb5kdc[16479](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:24:48 dist-test-slave-47qt krb5kdc[16479](info): set up 2 sockets
Jun 18 23:24:48 dist-test-slave-47qt krb5kdc[16479](info): commencing operation
krb5kdc: starting...
W20260618 23:24:49.245720 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:24:49 dist-test-slave-47qt krb5kdc[16479](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:24:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:24:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.39.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
I20260618 23:24:49.365386 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:37693
--webserver_interface=127.8.39.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:46403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:37693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:49.446256 16492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:49.446455 16492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:49.446476 16492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:49.447824 16492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:24:49.447878 16492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:49.447893 16492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:49.447907 16492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:24:49.447919 16492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:24:49.449893 16492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:46403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:37693
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:37693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:49.450151 16492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:49.450407 16492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:49.455651 16500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:49.455857 16497 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:49.455909 16498 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:49.456101 16492 server_base.cc:1034] running on GCE node
I20260618 23:24:49.456341 16492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:49.456615 16492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:49.457851 16492 hybrid_clock.cc:648] HybridClock initialized: now 1781825089457817 us; error 51 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-18T23:24:49Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1282 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:24:49 dist-test-slave-47qt krb5kdc[16505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:24:49 dist-test-slave-47qt krb5kdc[16505](info): set up 2 sockets
Jun 18 23:24:49 dist-test-slave-47qt krb5kdc[16505](info): commencing operation
krb5kdc: starting...
W20260618 23:24:50.525811 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:24:50 dist-test-slave-47qt krb5kdc[16505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:24:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:24:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.39.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
I20260618 23:24:50.648161 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:43111
--webserver_interface=127.8.39.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:43493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:43111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:50.731359 16518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:50.731616 16518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:50.731652 16518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:50.732856 16518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:24:50.733148 16518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:50.733215 16518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:50.733271 16518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:24:50.733295 16518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:24:50.734867 16518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:43493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:43111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:43111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:50.735154 16518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:50.735385 16518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:50.740175 16524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:50.740200 16526 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:50.740252 16523 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:50.740377 16518 server_base.cc:1034] running on GCE node
I20260618 23:24:50.740651 16518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:50.740939 16518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:50.742080 16518 hybrid_clock.cc:648] HybridClock initialized: now 1781825090742053 us; error 44 us; skew 500 ppm
Jun 18 23:24:50 dist-test-slave-47qt krb5kdc[16505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825090, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:50.748968 16518 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:24:50.749445 16518 webserver.cc:460] Webserver started at http://127.8.39.190:40287/ using document root <none> and password file <none>
I20260618 23:24:50.749691 16518 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:50.749759 16518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:50.749915 16518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:50.750944 16518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "72a7f794a3eb43fc9adf2b5eddf84803"
format_stamp: "Formatted at 2026-06-18 23:24:50 on dist-test-slave-47qt"
server_key: "b4024059910a952398b80033b966f007"
server_key_iv: "97326ab5e1cd6b150b821be15540af0b"
server_key_version: "encryptionkey@0"
I20260618 23:24:50.751300 16518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "72a7f794a3eb43fc9adf2b5eddf84803"
format_stamp: "Formatted at 2026-06-18 23:24:50 on dist-test-slave-47qt"
server_key: "b4024059910a952398b80033b966f007"
server_key_iv: "97326ab5e1cd6b150b821be15540af0b"
server_key_version: "encryptionkey@0"
I20260618 23:24:50.753404 16518 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260618 23:24:50.754402 16533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:50.754850 16518 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:24:50.755036 16518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "72a7f794a3eb43fc9adf2b5eddf84803"
format_stamp: "Formatted at 2026-06-18 23:24:50 on dist-test-slave-47qt"
server_key: "b4024059910a952398b80033b966f007"
server_key_iv: "97326ab5e1cd6b150b821be15540af0b"
server_key_version: "encryptionkey@0"
I20260618 23:24:50.755245 16518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:50.776849 16518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:50.779152 16518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:50.779316 16518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:50.783519 16518 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:43111
I20260618 23:24:50.783551 16585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:43111 every 8 connection(s)
I20260618 23:24:50.784238 16518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:24:50.785230 16586 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:24:50.787536 16586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803: Bootstrap starting.
I20260618 23:24:50.788091 16586 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803: Neither blocks nor log segments found. Creating new log.
I20260618 23:24:50.788367 16586 log.cc:824] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:50.789111 16586 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803: No bootstrap required, opened a new log
I20260618 23:24:50.790671 16586 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a7f794a3eb43fc9adf2b5eddf84803" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 43111 } }
I20260618 23:24:50.790833 16586 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:24:50.790882 16586 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72a7f794a3eb43fc9adf2b5eddf84803, State: Initialized, Role: FOLLOWER
I20260618 23:24:50.791028 16586 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a7f794a3eb43fc9adf2b5eddf84803" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 43111 } }
I20260618 23:24:50.791107 16586 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:24:50.791144 16586 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:24:50.791193 16586 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:24:50.791684 16586 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a7f794a3eb43fc9adf2b5eddf84803" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 43111 } }
I20260618 23:24:50.791796 16586 leader_election.cc:302] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72a7f794a3eb43fc9adf2b5eddf84803; no voters:
I20260618 23:24:50.791967 16586 leader_election.cc:288] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:24:50.792004 16589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:24:50.792204 16586 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:24:50.792212 16589 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [term 1 LEADER]: Becoming Leader. State: Replica: 72a7f794a3eb43fc9adf2b5eddf84803, State: Running, Role: LEADER
I20260618 23:24:50.792392 16589 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a7f794a3eb43fc9adf2b5eddf84803" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 43111 } }
I20260618 23:24:50.792979 16590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72a7f794a3eb43fc9adf2b5eddf84803" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a7f794a3eb43fc9adf2b5eddf84803" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 43111 } } }
I20260618 23:24:50.792989 16591 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72a7f794a3eb43fc9adf2b5eddf84803. Latest consensus state: current_term: 1 leader_uuid: "72a7f794a3eb43fc9adf2b5eddf84803" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a7f794a3eb43fc9adf2b5eddf84803" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 43111 } } }
I20260618 23:24:50.793078 16590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:50.793097 16591 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:50.793285 16598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:24:50.793797 16598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:24:50.793797 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 16518
I20260618 23:24:50.793901 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:24:50.794102 8350 external_mini_cluster.cc:1428] Setting key 9e286a73bb20bf09b2922a19934cda2d
I20260618 23:24:50.795500 16598 catalog_manager.cc:1347] Generated new cluster ID: b4c6152759fc43148cfed294e0e8295b
I20260618 23:24:50.795552 16598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 18 23:24:50 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825090, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:50.803699 16598 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:24:50.803979 16608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:50.794564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40248 (local address 127.8.39.190:43111)
0618 23:24:50.794711 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:50.794720 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:50.794815 (+ 95us) server_negotiation.cc:407] Connection header received
0618 23:24:50.794958 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:50.794966 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:50.795023 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:50.795183 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:50.795920 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:50.796831 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:50.797663 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:50.798079 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:50.798193 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:50.800466 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:50.800470 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:50.800470 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:50.800482 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:50.802230 (+ 1748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:50.802618 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:50.802622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:50.802622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:50.802735 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:50.802943 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:50.802944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:50.802945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:50.803186 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:50.803531 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:50.803606 (+ 75us) server_negotiation.cc:299] Negotiation successful
0618 23:24:50.803771 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":71,"threads_started":1}
I20260618 23:24:50.804354 16598 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:24:50.807452 16598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803: Generated new TSK 0
I20260618 23:24:50.807612 16598 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.39.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
I20260618 23:24:50.840750 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.129:0
--local_ip_for_outbound_sockets=127.8.39.129
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:43111
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:43493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:50.943029 16612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:50.943224 16612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:50.943248 16612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:50.944391 16612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:50.944438 16612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:50.944466 16612 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.129
I20260618 23:24:50.945978 16612 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:43493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:43111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.129
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:50.946198 16612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:50.946437 16612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:50.951535 16617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:50.951548 16620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:50.951578 16618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:50.951901 16612 server_base.cc:1034] running on GCE node
I20260618 23:24:50.952126 16612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:50.952415 16612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:50.953593 16612 hybrid_clock.cc:648] HybridClock initialized: now 1781825090953588 us; error 40 us; skew 500 ppm
Jun 18 23:24:50 dist-test-slave-47qt krb5kdc[16505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825090, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:50.956739 16612 init.cc:377] Logged in from keytab as kudu/127.8.39.129@KRBTEST.COM (short username kudu)
I20260618 23:24:50.957254 16612 webserver.cc:460] Webserver started at http://127.8.39.129:34395/ using document root <none> and password file <none>
I20260618 23:24:50.957487 16612 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:50.957554 16612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:50.957686 16612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:50.958756 16612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-0/data/instance:
uuid: "6ec4a4862163468ba35a276d05293f18"
format_stamp: "Formatted at 2026-06-18 23:24:50 on dist-test-slave-47qt"
server_key: "9dbdf67d922ca311aac425ba43ee56c0"
server_key_iv: "70943cadff2c07ad0b53047e6426c484"
server_key_version: "encryptionkey@0"
I20260618 23:24:50.959161 16612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance:
uuid: "6ec4a4862163468ba35a276d05293f18"
format_stamp: "Formatted at 2026-06-18 23:24:50 on dist-test-slave-47qt"
server_key: "9dbdf67d922ca311aac425ba43ee56c0"
server_key_iv: "70943cadff2c07ad0b53047e6426c484"
server_key_version: "encryptionkey@0"
I20260618 23:24:50.960700 16612 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:24:50.961848 16627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:50.962281 16612 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:50.962482 16612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-0/wal
uuid: "6ec4a4862163468ba35a276d05293f18"
format_stamp: "Formatted at 2026-06-18 23:24:50 on dist-test-slave-47qt"
server_key: "9dbdf67d922ca311aac425ba43ee56c0"
server_key_iv: "70943cadff2c07ad0b53047e6426c484"
server_key_version: "encryptionkey@0"
I20260618 23:24:50.962786 16612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:50.975833 16612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:50.978128 16612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:50.978288 16612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:50.978521 16612 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:50.978828 16612 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:50.978890 16612 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:50.978930 16612 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:50.978957 16612 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:50.984741 16612 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.129:40217
I20260618 23:24:50.984800 16740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.129:40217 every 8 connection(s)
I20260618 23:24:50.985152 16612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
I20260618 23:24:50.986418 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 16612
I20260618 23:24:50.986553 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance
I20260618 23:24:50.986740 8350 external_mini_cluster.cc:1428] Setting key b797dc57b806893b80ee0f9069c47cea
Jun 18 23:24:50 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825090, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.39.130@KRBTEST.COM; defaulting to no policy
I20260618 23:24:50.997833 16608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:50.986002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:53423 (local address 127.8.39.190:43111)
0618 23:24:50.986127 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:50.986129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:50.986547 (+ 418us) server_negotiation.cc:407] Connection header received
0618 23:24:50.987493 (+ 946us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:50.987494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:50.987508 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:50.987559 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:50.988656 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:50.989500 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:50.990498 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:50.990838 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:50.990921 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:50.993458 (+ 2537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:50.993460 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:50.993460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:50.993468 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:50.995833 (+ 2365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:50.996388 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:50.996390 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:50.996390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:50.996461 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:50.996728 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:50.996729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:50.996729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:50.996857 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:50.996926 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:50.997688 (+ 762us) server_negotiation.cc:299] Negotiation successful
0618 23:24:50.997764 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260618 23:24:50.998135 16743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:50.986004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.129:53423)
0618 23:24:50.986390 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:50.986431 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:50.987363 (+ 932us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:50.987625 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:50.987629 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:50.988342 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:50.988534 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:50.988538 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:50.989607 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:50.989608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:50.990402 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:50.990405 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:50.990966 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:50.990967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:50.991071 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:50.991663 (+ 592us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:50.991679 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:50.993317 (+ 1638us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:50.996004 (+ 2687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:50.996011 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:50.996013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:50.996270 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:50.996539 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:50.996540 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:50.996540 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:50.996641 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:50.997003 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:50.997018 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:50.997365 (+ 347us) client_negotiation.cc:769] Sending connection context
0618 23:24:50.997675 (+ 310us) client_negotiation.cc:240] Negotiation successful
0618 23:24:50.997920 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":290,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.130@KRBTEST.COM" created.
I20260618 23:24:50.998894 16741 heartbeater.cc:347] Connected to a master server at 127.8.39.190:43111
I20260618 23:24:50.999007 16741 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:50.999210 16741 heartbeater.cc:511] Master 127.8.39.190:43111 requested a full tablet report, sending...
I20260618 23:24:50.999864 16550 ts_manager.cc:194] Registered new tserver with Master: 6ec4a4862163468ba35a276d05293f18 (127.8.39.129:40217)
I20260618 23:24:51.000506 16550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.129@KRBTEST.COM'} at 127.8.39.129:53423
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
I20260618 23:24:51.016057 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.130:0
--local_ip_for_outbound_sockets=127.8.39.130
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:43111
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:43493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:51.103641 16746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:51.103855 16746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:51.103883 16746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:51.105096 16746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:51.105170 16746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:51.105197 16746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.130
I20260618 23:24:51.106827 16746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:43493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:43111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.130
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:51.107282 16746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:51.107568 16746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:51.113853 16752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:51.113927 16746 server_base.cc:1034] running on GCE node
W20260618 23:24:51.113853 16754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:51.113921 16751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:51.114302 16746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:51.114548 16746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:51.115739 16746 hybrid_clock.cc:648] HybridClock initialized: now 1781825091115716 us; error 27 us; skew 500 ppm
Jun 18 23:24:51 dist-test-slave-47qt krb5kdc[16505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825091, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:51.118274 16746 init.cc:377] Logged in from keytab as kudu/127.8.39.130@KRBTEST.COM (short username kudu)
I20260618 23:24:51.118692 16746 webserver.cc:460] Webserver started at http://127.8.39.130:40617/ using document root <none> and password file <none>
I20260618 23:24:51.118913 16746 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:51.118965 16746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:51.119093 16746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:51.120146 16746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-1/data/instance:
uuid: "10fb7754ca0c4fbfa10d42bb9b0914b8"
format_stamp: "Formatted at 2026-06-18 23:24:51 on dist-test-slave-47qt"
server_key: "c8b0be29f2694453b01a5e2138c7e260"
server_key_iv: "dc51cac7da6d8172a96c58a09e1b92e6"
server_key_version: "encryptionkey@0"
I20260618 23:24:51.120501 16746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance:
uuid: "10fb7754ca0c4fbfa10d42bb9b0914b8"
format_stamp: "Formatted at 2026-06-18 23:24:51 on dist-test-slave-47qt"
server_key: "c8b0be29f2694453b01a5e2138c7e260"
server_key_iv: "dc51cac7da6d8172a96c58a09e1b92e6"
server_key_version: "encryptionkey@0"
I20260618 23:24:51.122155 16746 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:24:51.123082 16761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:51.123349 16746 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:51.123510 16746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-1/wal
uuid: "10fb7754ca0c4fbfa10d42bb9b0914b8"
format_stamp: "Formatted at 2026-06-18 23:24:51 on dist-test-slave-47qt"
server_key: "c8b0be29f2694453b01a5e2138c7e260"
server_key_iv: "dc51cac7da6d8172a96c58a09e1b92e6"
server_key_version: "encryptionkey@0"
I20260618 23:24:51.123696 16746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:51.138932 16746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:51.141686 16746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:51.141832 16746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:51.142048 16746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:51.142357 16746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:51.142417 16746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:51.142449 16746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:51.142469 16746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:51.149235 16746 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.130:33417
I20260618 23:24:51.149287 16874 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.130:33417 every 8 connection(s)
I20260618 23:24:51.149622 16746 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
I20260618 23:24:51.151484 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 16746
I20260618 23:24:51.151638 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance
I20260618 23:24:51.151858 8350 external_mini_cluster.cc:1428] Setting key e29a9403d8436e799a30740b12edc84a
Jun 18 23:24:51 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825091, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:51.161203 16608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:51.150398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:45259 (local address 127.8.39.190:43111)
0618 23:24:51.150489 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:51.150492 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:51.150675 (+ 183us) server_negotiation.cc:407] Connection header received
0618 23:24:51.151145 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:51.151146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:51.151156 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:51.151217 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:51.152388 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.152989 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.154921 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.155140 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.155231 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:51.157891 (+ 2660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:51.157892 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:51.157893 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:51.157901 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:51.159651 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:51.160139 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:51.160141 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:51.160141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:51.160207 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:51.160457 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:51.160458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:51.160459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:51.160605 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:51.160678 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:51.160977 (+ 299us) server_negotiation.cc:299] Negotiation successful
0618 23:24:51.161073 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260618 23:24:51.161468 16877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:51.150371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.130:45259)
0618 23:24:51.150546 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:51.150587 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:51.151058 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:51.151293 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:51.151298 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:51.152110 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:51.152297 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:51.152302 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.153086 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.153088 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:51.154808 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:51.154811 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.155239 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.155240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:51.155369 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:51.155927 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:51.155937 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:51.157798 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:51.159759 (+ 1961us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:51.159765 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:51.159768 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:51.160050 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:51.160285 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:51.160286 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:51.160286 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:51.160385 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:51.160702 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:51.160708 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:51.160828 (+ 120us) client_negotiation.cc:769] Sending connection context
0618 23:24:51.160923 (+ 95us) client_negotiation.cc:240] Negotiation successful
0618 23:24:51.161057 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":104,"threads_started":1}
I20260618 23:24:51.161953 16875 heartbeater.cc:347] Connected to a master server at 127.8.39.190:43111
WARNING: no policy specified for kudu/127.8.39.131@KRBTEST.COM; defaulting to no policy
I20260618 23:24:51.162058 16875 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:51.162231 16875 heartbeater.cc:511] Master 127.8.39.190:43111 requested a full tablet report, sending...
I20260618 23:24:51.162596 16550 ts_manager.cc:194] Registered new tserver with Master: 10fb7754ca0c4fbfa10d42bb9b0914b8 (127.8.39.130:33417)
I20260618 23:24:51.163065 16550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.130@KRBTEST.COM'} at 127.8.39.130:45259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
I20260618 23:24:51.180617 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.131:0
--local_ip_for_outbound_sockets=127.8.39.131
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:43111
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:43493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:51.265913 16880 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:51.266113 16880 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:51.266137 16880 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:51.267935 16880 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:51.268036 16880 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:51.268081 16880 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.131
I20260618 23:24:51.270722 16880 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:43493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:43111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16880
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.131
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:51.270938 16880 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:51.271118 16880 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:51.276064 16888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:51.276101 16886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:51.276103 16885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:51.276316 16880 server_base.cc:1034] running on GCE node
I20260618 23:24:51.276516 16880 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:51.276726 16880 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:51.277859 16880 hybrid_clock.cc:648] HybridClock initialized: now 1781825091277846 us; error 28 us; skew 500 ppm
Jun 18 23:24:51 dist-test-slave-47qt krb5kdc[16505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825091, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:51.280442 16880 init.cc:377] Logged in from keytab as kudu/127.8.39.131@KRBTEST.COM (short username kudu)
I20260618 23:24:51.280853 16880 webserver.cc:460] Webserver started at http://127.8.39.131:35335/ using document root <none> and password file <none>
I20260618 23:24:51.281096 16880 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:51.281186 16880 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:51.281345 16880 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:24:51.282410 16880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-2/data/instance:
uuid: "84def92eb3ee4cf2bd97f3e92415c498"
format_stamp: "Formatted at 2026-06-18 23:24:51 on dist-test-slave-47qt"
server_key: "0d78114a2ff6d2a2b50223126c9776a5"
server_key_iv: "35fc751552d4bab4792d3e8938fd38be"
server_key_version: "encryptionkey@0"
I20260618 23:24:51.282768 16880 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance:
uuid: "84def92eb3ee4cf2bd97f3e92415c498"
format_stamp: "Formatted at 2026-06-18 23:24:51 on dist-test-slave-47qt"
server_key: "0d78114a2ff6d2a2b50223126c9776a5"
server_key_iv: "35fc751552d4bab4792d3e8938fd38be"
server_key_version: "encryptionkey@0"
I20260618 23:24:51.284068 16880 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:24:51.284857 16895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:51.285043 16880 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:51.285153 16880 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-2/wal
uuid: "84def92eb3ee4cf2bd97f3e92415c498"
format_stamp: "Formatted at 2026-06-18 23:24:51 on dist-test-slave-47qt"
server_key: "0d78114a2ff6d2a2b50223126c9776a5"
server_key_iv: "35fc751552d4bab4792d3e8938fd38be"
server_key_version: "encryptionkey@0"
I20260618 23:24:51.285243 16880 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:51.301604 16880 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:51.304620 16880 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:51.304795 16880 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:51.305042 16880 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:51.305419 16880 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:51.305478 16880 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:51.305522 16880 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:51.305549 16880 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:51.311873 16880 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.131:37451
I20260618 23:24:51.311915 17008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.131:37451 every 8 connection(s)
I20260618 23:24:51.312253 16880 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
I20260618 23:24:51.316612 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 16880
I20260618 23:24:51.316758 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance
I20260618 23:24:51.316990 8350 external_mini_cluster.cc:1428] Setting key 27523b6005dcf8889f28093846bd5c8f
Jun 18 23:24:51 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825091, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:24:51.322598 16608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:51.313031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:44297 (local address 127.8.39.190:43111)
0618 23:24:51.313092 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:51.313094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:51.313263 (+ 169us) server_negotiation.cc:407] Connection header received
0618 23:24:51.313743 (+ 480us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:51.313744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:51.313752 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:51.313826 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:51.314756 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.315260 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.316358 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.316771 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.316859 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:51.319504 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:51.319505 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:51.319506 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:51.319512 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:51.320946 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:51.321501 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:51.321502 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:51.321503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:51.321559 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:51.321949 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:51.321950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:51.321950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:51.322066 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:51.322135 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:51.322463 (+ 328us) server_negotiation.cc:299] Negotiation successful
0618 23:24:51.322534 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260618 23:24:51.322708 17011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:51.312948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.131:44297)
0618 23:24:51.313104 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:51.313179 (+ 75us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:51.313660 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:51.313880 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:51.313883 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:51.314440 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:51.314637 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:51.314641 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.315371 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.315371 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:51.316191 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:51.316194 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.316886 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.316887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:51.317004 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:51.317668 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:51.317679 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:51.319421 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:24:51.321165 (+ 1744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:51.321172 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:51.321175 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:51.321413 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:51.321709 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:24:51.321709 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:24:51.321710 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:24:51.321845 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:24:51.322149 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:24:51.322154 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:24:51.322273 (+ 119us) client_negotiation.cc:769] Sending connection context
0618 23:24:51.322376 (+ 103us) client_negotiation.cc:240] Negotiation successful
0618 23:24:51.322500 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":93,"threads_started":1}
I20260618 23:24:51.323390 17009 heartbeater.cc:347] Connected to a master server at 127.8.39.190:43111
I20260618 23:24:51.323510 17009 heartbeater.cc:464] Registering TS with master...
I20260618 23:24:51.323706 17009 heartbeater.cc:511] Master 127.8.39.190:43111 requested a full tablet report, sending...
I20260618 23:24:51.324395 16550 ts_manager.cc:194] Registered new tserver with Master: 84def92eb3ee4cf2bd97f3e92415c498 (127.8.39.131:37451)
I20260618 23:24:51.324739 16550 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.131@KRBTEST.COM'} at 127.8.39.131:44297
I20260618 23:24:51.329788 8350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.39.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.39.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.39.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.129.keytab.
Entry for principal oryx/127.8.39.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.129.keytab.
WARNING: no policy specified for oryx/127.8.39.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.39.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.39.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.130.keytab.
Entry for principal oryx/127.8.39.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.130.keytab.
WARNING: no policy specified for oryx/127.8.39.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.39.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.39.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.131.keytab.
Entry for principal oryx/127.8.39.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.131.keytab.
I20260618 23:24:51.404768 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 16612
I20260618 23:24:51.409307 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 16746
I20260618 23:24:51.413784 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 16880
I20260618 23:24:51.418206 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 16518
I20260618 23:24:51.423219 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:43111
--webserver_interface=127.8.39.190
--webserver_port=40287
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:43493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:43111
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:24:51.503855 17018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:51.504112 17018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:51.504151 17018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:51.505300 17018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:24:51.505373 17018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:51.505398 17018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:51.505427 17018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:24:51.505451 17018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:24:51.506939 17018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:43493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:43111
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:43111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=40287
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:51.507210 17018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:51.507436 17018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:51.512230 17024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:51.512318 17023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:51.512238 17026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:51.512579 17018 server_base.cc:1034] running on GCE node
I20260618 23:24:51.512784 17018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:51.513021 17018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:51.514287 17018 hybrid_clock.cc:648] HybridClock initialized: now 1781825091514264 us; error 39 us; skew 500 ppm
Jun 18 23:24:51 dist-test-slave-47qt krb5kdc[16505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825091, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:51.517244 17018 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:24:51.517704 17018 webserver.cc:460] Webserver started at http://127.8.39.190:40287/ using document root <none> and password file <none>
I20260618 23:24:51.517904 17018 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:51.517953 17018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:51.519224 17018 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:51.519869 17033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:51.520047 17018 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:51.520144 17018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "72a7f794a3eb43fc9adf2b5eddf84803"
format_stamp: "Formatted at 2026-06-18 23:24:50 on dist-test-slave-47qt"
server_key: "b4024059910a952398b80033b966f007"
server_key_iv: "97326ab5e1cd6b150b821be15540af0b"
server_key_version: "encryptionkey@0"
I20260618 23:24:51.520483 17018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:51.528252 17018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:51.530501 17018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:51.530659 17018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:51.534611 17018 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:43111
I20260618 23:24:51.534655 17085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:43111 every 8 connection(s)
I20260618 23:24:51.535030 17018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:24:51.536394 17086 sys_catalog.cc:263] Verifying existing consensus state
I20260618 23:24:51.537019 17086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803: Bootstrap starting.
I20260618 23:24:51.537879 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 17018
I20260618 23:24:51.538267 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.129:40217
--local_ip_for_outbound_sockets=127.8.39.129
--tserver_master_addrs=127.8.39.190:43111
--webserver_port=34395
--webserver_interface=127.8.39.129
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:43493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.39.129
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:24:51.545797 17086 log.cc:824] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803: Log is configured to *not* fsync() on all Append() calls
I20260618 23:24:51.547413 17086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 23:24:51.547704 17086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803: Bootstrap complete.
I20260618 23:24:51.551086 17086 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a7f794a3eb43fc9adf2b5eddf84803" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 43111 } }
I20260618 23:24:51.551355 17086 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72a7f794a3eb43fc9adf2b5eddf84803, State: Initialized, Role: FOLLOWER
I20260618 23:24:51.551533 17086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a7f794a3eb43fc9adf2b5eddf84803" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 43111 } }
I20260618 23:24:51.551604 17086 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:24:51.551653 17086 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:24:51.551700 17086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [term 1 FOLLOWER]: Advancing to term 2
I20260618 23:24:51.552755 17086 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a7f794a3eb43fc9adf2b5eddf84803" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 43111 } }
I20260618 23:24:51.552867 17086 leader_election.cc:302] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72a7f794a3eb43fc9adf2b5eddf84803; no voters:
I20260618 23:24:51.553066 17086 leader_election.cc:288] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260618 23:24:51.553100 17090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [term 2 FOLLOWER]: Leader election won for term 2
I20260618 23:24:51.553251 17090 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [term 2 LEADER]: Becoming Leader. State: Replica: 72a7f794a3eb43fc9adf2b5eddf84803, State: Running, Role: LEADER
I20260618 23:24:51.553340 17090 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a7f794a3eb43fc9adf2b5eddf84803" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 43111 } }
I20260618 23:24:51.553347 17086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:24:51.553566 17091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "72a7f794a3eb43fc9adf2b5eddf84803" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a7f794a3eb43fc9adf2b5eddf84803" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 43111 } } }
I20260618 23:24:51.553630 17091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:51.553771 17092 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72a7f794a3eb43fc9adf2b5eddf84803. Latest consensus state: current_term: 2 leader_uuid: "72a7f794a3eb43fc9adf2b5eddf84803" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72a7f794a3eb43fc9adf2b5eddf84803" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 43111 } } }
I20260618 23:24:51.553822 17092 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803 [sys.catalog]: This master's current role is: LEADER
I20260618 23:24:51.553930 17094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:24:51.554379 17094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:24:51.554512 17094 catalog_manager.cc:1259] Loaded cluster ID: b4c6152759fc43148cfed294e0e8295b
I20260618 23:24:51.554549 17094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:24:51.555683 17094 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:24:51.555858 17094 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 72a7f794a3eb43fc9adf2b5eddf84803: Loaded TSK: 0
I20260618 23:24:51.556062 17094 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260618 23:24:51.633528 17088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:51.633745 17088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:51.633787 17088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:51.635008 17088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:51.635077 17088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:51.635124 17088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.129
I20260618 23:24:51.636559 17088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:43493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.129.keytab
--principal=oryx/127.8.39.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.129:40217
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.39.129
--webserver_port=34395
--tserver_master_addrs=127.8.39.190:43111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.129
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:51.636819 17088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:51.637054 17088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:51.642041 17115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:51.642040 17112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:51.642156 17113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:51.642371 17088 server_base.cc:1034] running on GCE node
I20260618 23:24:51.642678 17088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:51.643014 17088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:51.644311 17088 hybrid_clock.cc:648] HybridClock initialized: now 1781825091644270 us; error 47 us; skew 500 ppm
Jun 18 23:24:51 dist-test-slave-47qt krb5kdc[16505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825091, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:51.647733 17088 init.cc:377] Logged in from keytab as oryx/127.8.39.129@KRBTEST.COM (short username oryx)
I20260618 23:24:51.648281 17088 webserver.cc:460] Webserver started at http://127.8.39.129:34395/ using document root <none> and password file <none>
I20260618 23:24:51.648557 17088 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:51.648639 17088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:51.650090 17088 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:51.650699 17122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:51.650966 17088 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:51.651067 17088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-0/wal
uuid: "6ec4a4862163468ba35a276d05293f18"
format_stamp: "Formatted at 2026-06-18 23:24:50 on dist-test-slave-47qt"
server_key: "9dbdf67d922ca311aac425ba43ee56c0"
server_key_iv: "70943cadff2c07ad0b53047e6426c484"
server_key_version: "encryptionkey@0"
I20260618 23:24:51.651490 17088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:51.665748 17088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:51.668416 17088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:51.668567 17088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:51.668805 17088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:51.669173 17088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:51.669219 17088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:51.669250 17088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:51.669279 17088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:51.676607 17088 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.129:40217
I20260618 23:24:51.676723 17235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.129:40217 every 8 connection(s)
I20260618 23:24:51.677079 17088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
I20260618 23:24:51.684751 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 17088
I20260618 23:24:51.685313 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.130:33417
--local_ip_for_outbound_sockets=127.8.39.130
--tserver_master_addrs=127.8.39.190:43111
--webserver_port=40617
--webserver_interface=127.8.39.130
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:43493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.39.130
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 23:24:51 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:51 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:24:51.687389 17238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:51.677815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.129:45401)
0618 23:24:51.678019 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:51.678065 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:51.678666 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:51.679130 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:51.679134 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:51.679890 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:51.680148 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:51.680153 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.681569 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.681570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:51.682838 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:51.682841 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.683456 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.683457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:51.683597 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:51.684334 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:51.684349 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:51.687182 (+ 2833us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":113,"threads_started":1}
W20260618 23:24:51.687485 17239 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:51.677959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:45401 (local address 127.8.39.190:43111)
0618 23:24:51.678122 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:51.678138 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:51.678167 (+ 29us) server_negotiation.cc:407] Connection header received
0618 23:24:51.678880 (+ 713us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:51.678885 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:51.678944 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:51.679053 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:51.680265 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.681468 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.682943 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.683360 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.683447 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:51.687295 (+ 3848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.129:45401: BlockingRecv error: recv got EOF from 127.8.39.129:45401 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":72,"threads_started":1}
W20260618 23:24:51.687650 17236 heartbeater.cc:644] Failed to heartbeat to 127.8.39.190:43111 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.39.190:43111: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Jun 18 23:24:51 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:51 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:24:51.693423 17238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:51.687867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.129:42417)
0618 23:24:51.687909 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:51.687919 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:51.687959 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:51.688118 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:51.688119 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:51.688268 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:51.688322 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:51.688323 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.689224 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.689226 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:51.690149 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:51.690152 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.690613 (+ 461us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.690614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:51.690691 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:51.691179 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:51.691185 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:51.693353 (+ 2168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260618 23:24:51.693536 17239 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:51.687914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:42417 (local address 127.8.39.190:43111)
0618 23:24:51.687947 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:51.687949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:51.687957 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:24:51.688019 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:51.688021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:51.688034 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:51.688102 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:51.688407 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.689110 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.690246 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.690526 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.690605 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:51.693460 (+ 2855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.129:42417: BlockingRecv error: recv got EOF from 127.8.39.129:42417 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Jun 18 23:24:51 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:51 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:24:51.702338 17238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:51.693955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.129:38509)
0618 23:24:51.694000 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:51.694010 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:51.694046 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:51.697337 (+ 3291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:51.697339 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:51.697479 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:51.697514 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:51.697516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.698429 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.698430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:51.699224 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:51.699226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.699685 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.699687 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:51.699744 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:51.700234 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:51.700239 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:51.702272 (+ 2033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":30}
W20260618 23:24:51.702430 17239 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:51.694023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:38509 (local address 127.8.39.190:43111)
0618 23:24:51.697211 (+ 3188us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:51.697213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:51.697222 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:24:51.697245 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:51.697245 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:51.697256 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:51.697320 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:51.697594 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.698343 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.699306 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.699603 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.699668 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:51.702369 (+ 2701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.129:38509: BlockingRecv error: recv got EOF from 127.8.39.129:38509 (error 108)
Metrics: {"server-negotiator.queue_time_us":3165}
W20260618 23:24:51.702896 17236 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260618 23:24:51.770510 17240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:51.770737 17240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:51.770778 17240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:51.771919 17240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:51.771982 17240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:51.772028 17240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.130
I20260618 23:24:51.773536 17240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:43493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.130.keytab
--principal=oryx/127.8.39.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.130:33417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.39.130
--webserver_port=40617
--tserver_master_addrs=127.8.39.190:43111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.130
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:51.773806 17240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:51.774029 17240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:51.779126 17246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:51.779161 17248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:51.779161 17245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:51.779454 17240 server_base.cc:1034] running on GCE node
I20260618 23:24:51.779651 17240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:51.779942 17240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:51.781103 17240 hybrid_clock.cc:648] HybridClock initialized: now 1781825091781099 us; error 37 us; skew 500 ppm
Jun 18 23:24:51 dist-test-slave-47qt krb5kdc[16505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825091, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:51.783561 17240 init.cc:377] Logged in from keytab as oryx/127.8.39.130@KRBTEST.COM (short username oryx)
I20260618 23:24:51.784081 17240 webserver.cc:460] Webserver started at http://127.8.39.130:40617/ using document root <none> and password file <none>
I20260618 23:24:51.784323 17240 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:51.784407 17240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:51.786666 17240 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:51.787529 17255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:51.787705 17240 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260618 23:24:51.787808 17240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-1/wal
uuid: "10fb7754ca0c4fbfa10d42bb9b0914b8"
format_stamp: "Formatted at 2026-06-18 23:24:51 on dist-test-slave-47qt"
server_key: "c8b0be29f2694453b01a5e2138c7e260"
server_key_iv: "dc51cac7da6d8172a96c58a09e1b92e6"
server_key_version: "encryptionkey@0"
I20260618 23:24:51.788174 17240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:51.797106 17240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:51.799156 17240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:51.799317 17240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:51.799556 17240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:51.799852 17240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:51.799906 17240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:51.799947 17240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:51.799974 17240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:51.805922 17240 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.130:33417
I20260618 23:24:51.805985 17368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.130:33417 every 8 connection(s)
I20260618 23:24:51.806324 17240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
I20260618 23:24:51.810925 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 17240
I20260618 23:24:51.811244 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.131:37451
--local_ip_for_outbound_sockets=127.8.39.131
--tserver_master_addrs=127.8.39.190:43111
--webserver_port=35335
--webserver_interface=127.8.39.131
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:43493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.39.131
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 23:24:51 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:51 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
W20260618 23:24:51.813885 17239 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:51.807098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:42351 (local address 127.8.39.190:43111)
0618 23:24:51.807145 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:51.807147 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:51.807340 (+ 193us) server_negotiation.cc:407] Connection header received
0618 23:24:51.807802 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:51.807802 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:51.807815 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:51.807874 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:51.808740 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.809236 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.810168 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.810342 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.810393 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:51.813822 (+ 3429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.130:42351: BlockingRecv error: recv got EOF from 127.8.39.130:42351 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260618 23:24:51.813915 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:51.807062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.130:42351)
0618 23:24:51.807217 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:51.807257 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:51.807721 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:51.807921 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:51.807924 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:51.808470 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:51.808648 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:51.808652 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.809312 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.809313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:51.810092 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:51.810094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.810412 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.810412 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:51.810506 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:51.811070 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:51.811083 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:51.813673 (+ 2590us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
W20260618 23:24:51.814335 17369 heartbeater.cc:644] Failed to heartbeat to 127.8.39.190:43111 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.39.190:43111: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Jun 18 23:24:51 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:51 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:24:51.819166 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:51.814620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.130:46257)
0618 23:24:51.814662 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:51.814672 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:51.814731 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:51.814875 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:51.814876 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:51.815010 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:51.815051 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:51.815052 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.815675 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.815676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:51.816183 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:51.816184 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.816538 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.816539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:51.816603 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:51.817032 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:51.817037 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:51.819072 (+ 2035us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260618 23:24:51.819199 17239 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:51.814678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:46257 (local address 127.8.39.190:43111)
0618 23:24:51.814736 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:51.814739 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:51.814768 (+ 29us) server_negotiation.cc:407] Connection header received
0618 23:24:51.814792 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:51.814793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:51.814802 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:51.814878 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:51.815128 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.815583 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.816265 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.816452 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.816538 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:51.819157 (+ 2619us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.130:46257: BlockingRecv error: recv got EOF from 127.8.39.130:46257 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Jun 18 23:24:51 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:51 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:24:51.824031 17371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:51.819479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.130:37945)
0618 23:24:51.819555 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:51.819568 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:51.819628 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:51.819773 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:51.819774 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:51.819933 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:51.819970 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:51.819972 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.820576 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.820577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:51.821487 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:51.821490 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.821813 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.821814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:51.821878 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:51.822224 (+ 346us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:51.822228 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:51.823965 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":57}
W20260618 23:24:51.824168 17239 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:51.819519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:37945 (local address 127.8.39.190:43111)
0618 23:24:51.819555 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:51.819557 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:51.819612 (+ 55us) server_negotiation.cc:407] Connection header received
0618 23:24:51.819690 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:51.819690 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:51.819698 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:51.819742 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:51.820043 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.820498 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.821573 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.821741 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.821800 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:51.824093 (+ 2293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.130:37945: BlockingRecv error: recv got EOF from 127.8.39.130:37945 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260618 23:24:51.824316 17369 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260618 23:24:51.893899 17372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:24:51.894120 17372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:24:51.894160 17372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:24:51.895295 17372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:24:51.895361 17372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:24:51.895406 17372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.131
I20260618 23:24:51.896891 17372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:43493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/krb5kdc/oryx_127.8.39.131.keytab
--principal=oryx/127.8.39.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.131:37451
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.39.131
--webserver_port=35335
--tserver_master_addrs=127.8.39.190:43111
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.131
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:24:51.897173 17372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:24:51.897397 17372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:24:51.902243 17380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:51.902240 17378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:24:51.902248 17377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:24:51.902581 17372 server_base.cc:1034] running on GCE node
I20260618 23:24:51.902863 17372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:24:51.903100 17372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:24:51.904248 17372 hybrid_clock.cc:648] HybridClock initialized: now 1781825091904233 us; error 24 us; skew 500 ppm
Jun 18 23:24:51 dist-test-slave-47qt krb5kdc[16505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825091, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.39.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:24:51.907929 17372 init.cc:377] Logged in from keytab as oryx/127.8.39.131@KRBTEST.COM (short username oryx)
I20260618 23:24:51.908417 17372 webserver.cc:460] Webserver started at http://127.8.39.131:35335/ using document root <none> and password file <none>
I20260618 23:24:51.908643 17372 fs_manager.cc:359] Metadata directory not provided
I20260618 23:24:51.908735 17372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:24:51.910293 17372 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:51.911187 17387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:51.911402 17372 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:24:51.911502 17372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-2/wal
uuid: "84def92eb3ee4cf2bd97f3e92415c498"
format_stamp: "Formatted at 2026-06-18 23:24:51 on dist-test-slave-47qt"
server_key: "0d78114a2ff6d2a2b50223126c9776a5"
server_key_iv: "35fc751552d4bab4792d3e8938fd38be"
server_key_version: "encryptionkey@0"
I20260618 23:24:51.911883 17372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:24:51.924568 17372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:24:51.926883 17372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:24:51.927049 17372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:24:51.927289 17372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:24:51.927604 17372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:24:51.927661 17372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:51.927706 17372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:24:51.927736 17372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:24:51.934204 17372 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.131:37451
I20260618 23:24:51.934240 17500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.131:37451 every 8 connection(s)
I20260618 23:24:51.934590 17372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
I20260618 23:24:51.936869 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 17372
Jun 18 23:24:51 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:51 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
W20260618 23:24:51.943591 17239 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:51.935621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:36929 (local address 127.8.39.190:43111)
0618 23:24:51.935687 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:51.935690 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:51.935876 (+ 186us) server_negotiation.cc:407] Connection header received
0618 23:24:51.936403 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:51.936404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:51.936414 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:51.936466 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:51.937745 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.938519 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.939941 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.940144 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.940231 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:51.943545 (+ 3314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.131:36929: BlockingRecv error: recv got EOF from 127.8.39.131:36929 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
I20260618 23:24:51.943691 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:51.935626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.131:36929)
0618 23:24:51.935776 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:51.935809 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:51.936289 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:51.936524 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:51.936527 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:51.937351 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:51.937656 (+ 305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:51.937662 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.938620 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.938621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:51.939823 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:51.939825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.940242 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.940243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:51.940373 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:51.940883 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:51.940891 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:51.943480 (+ 2589us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
W20260618 23:24:51.944054 17501 heartbeater.cc:644] Failed to heartbeat to 127.8.39.190:43111 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.39.190:43111: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
I20260618 23:24:51.945338 17505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:51.937458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40250 (local address 127.8.39.190:43111)
0618 23:24:51.937577 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:51.937579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:51.937587 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:24:51.937608 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:51.937609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:51.937619 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:51.937694 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:51.938405 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.939075 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.940024 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.940325 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.940442 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:51.941951 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:24:51.941955 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:24:51.941956 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:24:51.941967 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:24:51.943880 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:51.944286 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:51.944290 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:51.944290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:51.944389 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:24:51.944605 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:24:51.944606 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:24:51.944606 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:24:51.944842 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:24:51.944993 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0618 23:24:51.945103 (+ 110us) server_negotiation.cc:299] Negotiation successful
0618 23:24:51.945232 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":58,"threads_started":1}
Jun 18 23:24:51 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:51 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:24:51.948678 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:51.944279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.131:58607)
0618 23:24:51.944370 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:51.944380 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:51.944422 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:51.944539 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:51.944539 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:51.944661 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:51.944703 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:51.944704 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.945590 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.945591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:51.946275 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:51.946277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.946643 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.946644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:51.946712 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:51.947126 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:51.947130 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:51.948601 (+ 1471us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":76}
W20260618 23:24:51.948725 17239 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:51.944330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:58607 (local address 127.8.39.190:43111)
0618 23:24:51.944368 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:51.944370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:51.944410 (+ 40us) server_negotiation.cc:407] Connection header received
0618 23:24:51.944475 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:51.944475 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:51.944484 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:51.944534 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:51.944783 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.945362 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.946364 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.946562 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.946644 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:51.948663 (+ 2019us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.131:58607: BlockingRecv error: recv got EOF from 127.8.39.131:58607 (error 108)
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":18}
Jun 18 23:24:51 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:51 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:24:51.952483 17503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:51.948953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.131:38693)
0618 23:24:51.948989 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:51.948998 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:51.949028 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:51.949191 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:51.949191 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:51.949301 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:51.949326 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:51.949327 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.949890 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.949891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:51.950347 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:51.950349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.950640 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.950640 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:51.950673 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:51.950980 (+ 307us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:51.950984 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:51.952444 (+ 1460us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":24}
W20260618 23:24:51.952548 17239 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:51.949014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:38693 (local address 127.8.39.190:43111)
0618 23:24:51.949046 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:51.949048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:51.949057 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:24:51.949074 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:51.949075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:51.949083 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:51.949187 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:51.949386 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.949791 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.950419 (+ 628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:51.950580 (+ 161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:51.950627 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:51.952513 (+ 1886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.131:38693: BlockingRecv error: recv got EOF from 127.8.39.131:38693 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260618 23:24:51.952608 17501 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 18 23:24:52 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:52 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:24:52.708750 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:52.703398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.129:37717)
0618 23:24:52.703544 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:52.703555 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:52.703614 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:52.703880 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:52.703881 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:52.704037 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:52.704082 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:52.704084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:52.704775 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:52.704782 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:52.705333 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:52.705334 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:52.705665 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:52.705666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:52.705707 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:52.706350 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:52.706356 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:52.708667 (+ 2311us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
W20260618 23:24:52.709226 17507 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:52.703501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:37717 (local address 127.8.39.190:43111)
0618 23:24:52.703627 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:52.703629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:52.703651 (+ 22us) server_negotiation.cc:407] Connection header received
0618 23:24:52.703674 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:52.703674 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:52.703685 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:52.703742 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:52.704183 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:52.704682 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:52.705408 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:52.705597 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:52.705654 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:52.709098 (+ 3444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.129:37717: BlockingRecv error: recv got EOF from 127.8.39.129:37717 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":71,"threads_started":1}
Jun 18 23:24:52 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:52 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:24:52.829746 17508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:52.824749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.130:58243)
0618 23:24:52.824903 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:52.824921 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:52.824975 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:52.825090 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:52.825090 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:52.825260 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:52.825315 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:52.825317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:52.826166 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:52.826167 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:52.827042 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:52.827045 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:52.827427 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:52.827428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:52.827473 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:52.828009 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:52.828014 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:52.829681 (+ 1667us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":100,"threads_started":1}
W20260618 23:24:52.829908 17507 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:52.824825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:58243 (local address 127.8.39.190:43111)
0618 23:24:52.824911 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:52.824913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:52.824976 (+ 63us) server_negotiation.cc:407] Connection header received
0618 23:24:52.825017 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:52.825018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:52.825030 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:52.825086 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:52.825412 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:52.826064 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:52.827129 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:52.827337 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:52.827399 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:52.829828 (+ 2429us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.130:58243: BlockingRecv error: recv got EOF from 127.8.39.130:58243 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Jun 18 23:24:52 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:52 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:24:52.957777 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:52.952998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.131:42669)
0618 23:24:52.953171 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:52.953184 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:52.953277 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:52.953458 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:52.953459 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:52.953690 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:52.953746 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:52.953749 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:52.954430 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:52.954431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:52.955037 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:52.955039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:52.955410 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:52.955411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:52.955468 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:52.955961 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:52.955966 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:52.957692 (+ 1726us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":101,"threads_started":1}
W20260618 23:24:52.957885 17507 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:52.953105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:42669 (local address 127.8.39.190:43111)
0618 23:24:52.953174 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:52.953176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:52.953258 (+ 82us) server_negotiation.cc:407] Connection header received
0618 23:24:52.953339 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:52.953340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:52.953352 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:52.953448 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:52.953849 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:52.954345 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:52.955122 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:52.955326 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:52.955409 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:52.957803 (+ 2394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.131:42669: BlockingRecv error: recv got EOF from 127.8.39.131:42669 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 18 23:24:53 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:53 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:24:53.714356 17510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:53.709556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.129:49671)
0618 23:24:53.709721 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:53.709735 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:53.709809 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:53.709973 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:53.709974 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:53.710129 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:53.710178 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:53.710180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:53.710993 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:53.710994 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:53.711580 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:53.711582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:53.711953 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:53.711954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:53.711991 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:53.712485 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:53.712489 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:53.714294 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":68,"threads_started":1}
W20260618 23:24:53.714463 17511 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:53.709693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:49671 (local address 127.8.39.190:43111)
0618 23:24:53.709815 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:53.709816 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:53.709826 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:24:53.709848 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:53.709848 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:53.709858 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:53.709916 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:53.710276 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:53.710797 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:53.711670 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:53.711863 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:53.711943 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:53.714387 (+ 2444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.129:49671: BlockingRecv error: recv got EOF from 127.8.39.129:49671 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":66,"threads_started":1}
Jun 18 23:24:53 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:53 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:24:53.834851 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:53.830207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.130:58797)
0618 23:24:53.830388 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:53.830413 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:53.830506 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:53.830778 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:53.830779 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:53.830955 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:53.830999 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:53.831001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:53.831663 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:53.831664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:53.832203 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:53.832205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:53.832578 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:53.832579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:53.832642 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:53.833107 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:53.833112 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:53.834752 (+ 1640us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
W20260618 23:24:53.834892 17511 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:53.830310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:58797 (local address 127.8.39.190:43111)
0618 23:24:53.830409 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:53.830411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:53.830488 (+ 77us) server_negotiation.cc:407] Connection header received
0618 23:24:53.830660 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:53.830662 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:53.830697 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:53.830770 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:53.831081 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:53.831579 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:53.832293 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:53.832491 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:53.832576 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:53.834830 (+ 2254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.130:58797: BlockingRecv error: recv got EOF from 127.8.39.130:58797 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Jun 18 23:24:53 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:53 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:24:53.963371 17513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:53.958340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.131:50595)
0618 23:24:53.958574 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:53.958585 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:53.958644 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:53.958780 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:53.958781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:53.958937 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:53.958983 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:53.958985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:53.959689 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:53.959690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:53.960280 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:53.960282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:53.960779 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:53.960780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:53.960846 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:53.961416 (+ 570us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:53.961422 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:53.963308 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":174,"threads_started":1}
W20260618 23:24:53.963519 17511 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:53.958502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:50595 (local address 127.8.39.190:43111)
0618 23:24:53.958581 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:53.958583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:53.958645 (+ 62us) server_negotiation.cc:407] Connection header received
0618 23:24:53.958689 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:53.958689 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:53.958701 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:53.958763 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:53.959047 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:53.959583 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:53.960367 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:53.960661 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:53.960755 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:53.963418 (+ 2663us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.131:50595: BlockingRecv error: recv got EOF from 127.8.39.131:50595 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 18 23:24:54 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:54 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:24:54.719467 17514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:54.714795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.129:50937)
0618 23:24:54.714945 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:54.714955 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:54.715016 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:54.715161 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:54.715162 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:54.715312 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:54.715357 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:54.715359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:54.716106 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:54.716107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:54.716671 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:54.716673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:54.717048 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:54.717049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:54.717087 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:54.717535 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:54.717540 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:54.719400 (+ 1860us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":93,"threads_started":1}
W20260618 23:24:54.719638 17515 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:54.714863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:50937 (local address 127.8.39.190:43111)
0618 23:24:54.714960 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:54.714962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:54.715001 (+ 39us) server_negotiation.cc:407] Connection header received
0618 23:24:54.715078 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:54.715079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:54.715089 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:54.715146 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:54.715433 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:54.716000 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:54.716772 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:54.716975 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:54.717038 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:54.719535 (+ 2497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.129:50937: BlockingRecv error: recv got EOF from 127.8.39.129:50937 (error 108)
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":46,"threads_started":1}
Jun 18 23:24:54 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:54 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:24:54.840055 17516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:54.835357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.130:44143)
0618 23:24:54.835594 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:54.835606 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:54.835697 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:54.835854 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:54.835855 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:54.836037 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:54.836080 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:54.836082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:54.836742 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:54.836743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:54.837344 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:54.837346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:54.837728 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:54.837729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:54.837790 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:54.838269 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:54.838274 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:54.839990 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":94,"threads_started":1}
W20260618 23:24:54.840167 17515 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:54.835505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:44143 (local address 127.8.39.190:43111)
0618 23:24:54.835587 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:54.835589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:54.835676 (+ 87us) server_negotiation.cc:407] Connection header received
0618 23:24:54.835758 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:54.835759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:54.835772 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:54.835847 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:54.836168 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:54.836660 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:54.837447 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:54.837645 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:54.837728 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:54.840072 (+ 2344us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.130:44143: BlockingRecv error: recv got EOF from 127.8.39.130:44143 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 18 23:24:54 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:54 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:24:54.968796 17517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:54.963843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.131:60157)
0618 23:24:54.963985 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:54.963995 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:54.964049 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:54.964179 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:54.964179 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:54.964411 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:54.964474 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:54.964477 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:54.965367 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:54.965368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:54.965921 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:54.965923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:54.966315 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:54.966316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:54.966373 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:54.966867 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:54.966881 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:54.968728 (+ 1847us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
W20260618 23:24:54.968925 17515 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:54.963929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:60157 (local address 127.8.39.190:43111)
0618 23:24:54.963989 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:54.963991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:54.964056 (+ 65us) server_negotiation.cc:407] Connection header received
0618 23:24:54.964093 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:54.964093 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:54.964105 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:54.964160 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:54.964563 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:54.965271 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:54.966023 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:54.966221 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:54.966314 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:54.968834 (+ 2520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.131:60157: BlockingRecv error: recv got EOF from 127.8.39.131:60157 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Jun 18 23:24:55 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:55 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:24:55.724685 17518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:55.719962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.129:33579)
0618 23:24:55.720094 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:55.720106 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:55.720162 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:55.720352 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:55.720353 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:55.720506 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:55.720552 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:55.720554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:55.721328 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:55.721328 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:55.721877 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:55.721878 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:55.722244 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:55.722245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:55.722282 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:55.722777 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:55.722782 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:55.724630 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
W20260618 23:24:55.724839 17519 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:55.720075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:33579 (local address 127.8.39.190:43111)
0618 23:24:55.720209 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:55.720211 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:55.720220 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:24:55.720252 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:55.720253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:55.720263 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:55.720344 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:55.720624 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:55.721246 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:55.721959 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:55.722174 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:55.722235 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:55.724725 (+ 2490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.129:33579: BlockingRecv error: recv got EOF from 127.8.39.129:33579 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
Jun 18 23:24:55 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:55 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:24:55.850514 17520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:55.840572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.130:35429)
0618 23:24:55.840711 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:55.840721 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:55.840769 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:55.840896 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:55.840897 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:55.841057 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:55.841104 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:55.841106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:55.841836 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:55.841836 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:55.842371 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:55.842372 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:55.842831 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:55.842832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:55.842901 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:55.847547 (+ 4646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:55.847556 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:55.850454 (+ 2898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":83,"threads_started":1}
W20260618 23:24:55.850677 17519 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:55.840645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:35429 (local address 127.8.39.190:43111)
0618 23:24:55.840733 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:55.840735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:55.840763 (+ 28us) server_negotiation.cc:407] Connection header received
0618 23:24:55.840814 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:55.840814 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:55.840824 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:55.840878 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:55.841246 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:55.841750 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:55.842458 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:55.842721 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:55.842816 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:55.850598 (+ 7782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.130:35429: BlockingRecv error: recv got EOF from 127.8.39.130:35429 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Jun 18 23:24:55 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:55 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:24:55.974344 17521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:55.969343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.131:54689)
0618 23:24:55.969489 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:55.969500 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:55.969567 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:55.969688 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:55.969689 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:55.969855 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:55.969910 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:55.969912 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:55.970552 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:55.970554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:55.971095 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:55.971098 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:55.971518 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:55.971519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:55.971571 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:55.972088 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:55.972093 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:55.974287 (+ 2194us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":97,"threads_started":1}
W20260618 23:24:55.974586 17519 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:55.969398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:54689 (local address 127.8.39.190:43111)
0618 23:24:55.969476 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:55.969478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:55.969566 (+ 88us) server_negotiation.cc:407] Connection header received
0618 23:24:55.969608 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:55.969609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:55.969620 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:55.969671 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:55.969970 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:55.970458 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:55.971168 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:55.971427 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:55.971495 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:55.974526 (+ 3031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.131:54689: BlockingRecv error: recv got EOF from 127.8.39.131:54689 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 18 23:24:56 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:56 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:24:56.730250 17522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:56.725224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.129:47963)
0618 23:24:56.725362 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:56.725373 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:56.725423 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:56.725608 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:56.725609 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:56.725774 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:56.725824 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:56.725826 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:56.726484 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:56.726485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:56.727062 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:56.727063 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:56.727451 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:56.727452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:56.727500 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:56.727974 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:56.727979 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:56.730182 (+ 2203us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":75,"threads_started":1}
W20260618 23:24:56.730615 17523 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:56.725318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:47963 (local address 127.8.39.190:43111)
0618 23:24:56.725456 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:56.725458 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:56.725466 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:24:56.725496 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:56.725496 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:56.725506 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:56.725568 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:56.725894 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:56.726398 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:56.727159 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:56.727374 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:56.727437 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:56.730565 (+ 3128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.129:47963: BlockingRecv error: recv got EOF from 127.8.39.129:47963 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":85,"threads_started":1}
Jun 18 23:24:56 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:56 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:24:56.855943 17524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:56.851039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.130:40535)
0618 23:24:56.851235 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:56.851253 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:56.851312 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:56.851446 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:56.851447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:56.851599 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:56.851646 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:56.851647 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:56.852301 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:56.852302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:56.852920 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:56.852923 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:56.853315 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:56.853316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:56.853358 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:56.853830 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:56.853835 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:56.855834 (+ 1999us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":80,"threads_started":1}
W20260618 23:24:56.855969 17523 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:56.851189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:40535 (local address 127.8.39.190:43111)
0618 23:24:56.851263 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:56.851265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:56.851297 (+ 32us) server_negotiation.cc:407] Connection header received
0618 23:24:56.851371 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:56.851372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:56.851383 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:56.851431 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:56.851705 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:56.852205 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:56.853006 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:56.853249 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:56.853305 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:56.855913 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.130:40535: BlockingRecv error: recv got EOF from 127.8.39.130:40535 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Jun 18 23:24:56 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:56 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:24:56.979250 17525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:56.974841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.131:42919)
0618 23:24:56.974975 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:56.974987 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:56.975051 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:56.975175 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:56.975176 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:56.975348 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:56.975396 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:56.975397 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:56.976027 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:56.976028 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:56.976554 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:56.976556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:56.976934 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:56.976935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:56.976977 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:56.977431 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:56.977435 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:56.979159 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
W20260618 23:24:56.979297 17523 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:56.974899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:42919 (local address 127.8.39.190:43111)
0618 23:24:56.974948 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:56.974951 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:56.975073 (+ 122us) server_negotiation.cc:407] Connection header received
0618 23:24:56.975105 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:56.975105 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:56.975117 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:56.975152 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:56.975459 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:56.975951 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:56.976661 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:56.976858 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:56.976920 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:56.979234 (+ 2314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.131:42919: BlockingRecv error: recv got EOF from 127.8.39.131:42919 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Jun 18 23:24:57 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:57 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:24:57.735638 17526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:57.730876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.129:41619)
0618 23:24:57.731094 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:57.731108 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:57.731192 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:57.731315 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:57.731316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:57.731476 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:57.731524 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:57.731525 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:57.732148 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:57.732149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:57.732662 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:57.732663 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:57.732991 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:57.732993 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:57.733050 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:57.733685 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:57.733690 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:57.735582 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":224,"threads_started":1}
W20260618 23:24:57.735772 17527 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:57.730930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:41619 (local address 127.8.39.190:43111)
0618 23:24:57.731047 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:57.731050 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:57.731184 (+ 134us) server_negotiation.cc:407] Connection header received
0618 23:24:57.731238 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:57.731238 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:57.731248 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:57.731303 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:57.731578 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:57.732081 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:57.732731 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:57.732910 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:57.732959 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:57.735682 (+ 2723us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.129:41619: BlockingRecv error: recv got EOF from 127.8.39.129:41619 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":70,"threads_started":1}
Jun 18 23:24:57 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:57 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:24:57.861719 17528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:57.856529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.130:55529)
0618 23:24:57.856722 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:57.856735 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:57.856839 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:57.857048 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:57.857049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:57.857287 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:57.857334 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:57.857337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:57.858068 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:57.858070 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:57.858667 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:57.858668 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:57.859104 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:57.859105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:57.859166 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:57.859715 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:57.859720 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:57.861619 (+ 1899us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":93,"threads_started":1}
W20260618 23:24:57.861765 17527 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:57.856634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:55529 (local address 127.8.39.190:43111)
0618 23:24:57.856722 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:57.856724 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:57.856816 (+ 92us) server_negotiation.cc:407] Connection header received
0618 23:24:57.856925 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:57.856925 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:57.856941 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:57.857004 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:57.857420 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:57.857975 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:57.858770 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:57.858988 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:57.859104 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:57.861698 (+ 2594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.130:55529: BlockingRecv error: recv got EOF from 127.8.39.130:55529 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 18 23:24:57 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:57 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:24:57.984443 17529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:57.979775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.131:38631)
0618 23:24:57.979913 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:57.979926 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:57.979998 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:57.980130 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:57.980131 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:57.980297 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:57.980344 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:57.980346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:57.981017 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:57.981018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:57.981659 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:57.981662 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:57.982013 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:57.982015 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:57.982082 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:57.982529 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:57.982533 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:57.984341 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":69,"threads_started":1}
W20260618 23:24:57.984496 17527 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:57.979822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:38631 (local address 127.8.39.190:43111)
0618 23:24:57.979891 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:57.979893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:57.979980 (+ 87us) server_negotiation.cc:407] Connection header received
0618 23:24:57.980045 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:57.980045 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:57.980064 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:57.980113 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:57.980409 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:57.980946 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:57.981736 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:57.981912 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:57.981997 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:57.984430 (+ 2433us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.131:38631: BlockingRecv error: recv got EOF from 127.8.39.131:38631 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 18 23:24:58 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:58 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:24:58.742264 17530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:58.736150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.129:37451)
0618 23:24:58.736282 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:58.736292 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:58.736347 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:58.736501 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:58.736502 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:58.736652 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:58.736699 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:58.736700 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:58.737749 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:58.737751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:58.738860 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:58.738863 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:58.739403 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:58.739404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:58.739458 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:58.739994 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:58.739999 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:58.742167 (+ 2168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":65,"threads_started":1}
W20260618 23:24:58.742313 17531 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:58.736184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:37451 (local address 127.8.39.190:43111)
0618 23:24:58.736314 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:58.736317 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:58.736332 (+ 15us) server_negotiation.cc:407] Connection header received
0618 23:24:58.736396 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:58.736397 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:58.736412 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:58.736500 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:58.736778 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:58.737637 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:58.738963 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:58.739295 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:58.739365 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:58.742245 (+ 2880us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.129:37451: BlockingRecv error: recv got EOF from 127.8.39.129:37451 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":46,"threads_started":1}
Jun 18 23:24:58 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:58 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:24:58.867187 17532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:58.862320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.130:46737)
0618 23:24:58.862479 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:58.862489 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:58.862538 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:58.862740 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:58.862742 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:58.862899 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:58.862947 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:58.862948 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:58.863708 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:58.863708 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:58.864573 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:58.864576 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:58.864953 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:58.864954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:58.864997 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:58.865499 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:58.865503 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:58.867131 (+ 1628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":98,"threads_started":1}
W20260618 23:24:58.867331 17531 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:58.862430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:46737 (local address 127.8.39.190:43111)
0618 23:24:58.862525 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:58.862527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:58.862536 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:24:58.862602 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:58.862603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:58.862613 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:58.862658 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:58.863097 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:58.863584 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:58.864689 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:58.864881 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:58.864947 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:58.867251 (+ 2304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.130:46737: BlockingRecv error: recv got EOF from 127.8.39.130:46737 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Jun 18 23:24:58 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:58 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:24:58.990036 17533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:58.984953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.131:54667)
0618 23:24:58.985078 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:58.985090 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:58.985214 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:58.985324 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:58.985325 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:58.985489 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:58.985542 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:58.985544 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:58.986170 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:58.986172 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:58.986709 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:58.986711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:58.987104 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:58.987105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:58.987158 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:58.987584 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:58.987588 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:58.989911 (+ 2323us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
W20260618 23:24:58.990046 17531 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:58.985010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:54667 (local address 127.8.39.190:43111)
0618 23:24:58.985101 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:58.985103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:58.985192 (+ 89us) server_negotiation.cc:407] Connection header received
0618 23:24:58.985269 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:58.985269 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:58.985281 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:58.985320 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:58.985605 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:58.986094 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:58.986804 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:58.986995 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:58.987053 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:58.989991 (+ 2938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.131:54667: BlockingRecv error: recv got EOF from 127.8.39.131:54667 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 18 23:24:59 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:59 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:24:59.747756 17534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:59.742940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.129:53569)
0618 23:24:59.743163 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:59.743194 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:59.743250 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:59.743563 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:59.743565 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:59.743750 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:59.743795 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:59.743796 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:59.744510 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:59.744511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:59.745035 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:59.745037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:59.745434 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:59.745435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:59.745492 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:59.745943 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:59.745947 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:59.747661 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":73,"threads_started":1}
W20260618 23:24:59.747797 17535 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:59.743227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:53569 (local address 127.8.39.190:43111)
0618 23:24:59.743378 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:59.743379 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:59.743388 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:24:59.743408 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:59.743409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:59.743438 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:59.743551 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:59.743879 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:59.744391 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:59.745149 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:59.745359 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:59.745436 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:59.747732 (+ 2296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.129:53569: BlockingRecv error: recv got EOF from 127.8.39.129:53569 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":63,"threads_started":1}
Jun 18 23:24:59 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:59 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:24:59.873171 17536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:59.867647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.130:41037)
0618 23:24:59.867817 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:59.867827 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:59.867892 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:59.868026 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:59.868026 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:59.868182 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:59.868229 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:59.868231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:59.869083 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:59.869084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:59.869677 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:59.869679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:59.870072 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:59.870073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:59.870129 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:59.870659 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:59.870663 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:59.873051 (+ 2388us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":102,"threads_started":1}
W20260618 23:24:59.873282 17535 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:59.867727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:41037 (local address 127.8.39.190:43111)
0618 23:24:59.867815 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:59.867817 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:59.867892 (+ 75us) server_negotiation.cc:407] Connection header received
0618 23:24:59.867952 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:59.867953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:59.867963 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:59.868008 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:59.868327 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:59.869007 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:59.869769 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:59.869979 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:59.870044 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:59.873213 (+ 3169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.130:41037: BlockingRecv error: recv got EOF from 127.8.39.130:41037 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Jun 18 23:24:59 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:24:59 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:24:59.996588 17537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:24:59.990783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.131:56341)
0618 23:24:59.991009 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0618 23:24:59.991023 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:24:59.991115 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:24:59.991334 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:24:59.991336 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:24:59.991591 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:24:59.991654 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:59.991657 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:59.992686 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:59.992688 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:59.993278 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:24:59.993280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:24:59.993706 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:24:59.993708 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:24:59.993791 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:59.994344 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:24:59.994349 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:24:59.996513 (+ 2164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":118,"threads_started":1}
W20260618 23:24:59.996793 17535 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:24:59.990949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:56341 (local address 127.8.39.190:43111)
0618 23:24:59.991010 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0618 23:24:59.991012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:24:59.991161 (+ 149us) server_negotiation.cc:407] Connection header received
0618 23:24:59.991196 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:24:59.991197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:24:59.991214 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:24:59.991277 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:24:59.991785 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:59.992524 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:59.993380 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:24:59.993581 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:24:59.993646 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:24:59.996704 (+ 3058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.131:56341: BlockingRecv error: recv got EOF from 127.8.39.131:56341 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Jun 18 23:25:00 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:25:00 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:25:00.753108 17538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:00.748273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.129:41543)
0618 23:25:00.748429 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:00.748440 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:00.748494 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:00.748622 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:00.748623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:00.748781 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:25:00.748840 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:00.748841 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:00.749538 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:00.749539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:00.750077 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:00.750079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:00.750438 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:00.750439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:00.750488 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:00.751071 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:25:00.751075 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:25:00.753014 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":80,"threads_started":1}
W20260618 23:25:00.753185 17539 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:25:00.748381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:41543 (local address 127.8.39.190:43111)
0618 23:25:00.748503 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:00.748505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:00.748512 (+ 7us) server_negotiation.cc:407] Connection header received
0618 23:25:00.748535 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:00.748536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:00.748546 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:00.748610 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:00.748908 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:00.749442 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:00.750159 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:00.750352 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:00.750409 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:00.753088 (+ 2679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.129:41543: BlockingRecv error: recv got EOF from 127.8.39.129:41543 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":69,"threads_started":1}
Jun 18 23:25:00 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:25:00 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:25:00.883183 17540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:00.873735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.130:39479)
0618 23:25:00.873855 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:00.873866 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:00.873912 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:00.874275 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:00.874276 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:00.874444 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:25:00.874501 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:00.874503 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:00.875253 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:00.875254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:00.875989 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:00.875993 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:00.876377 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:00.876378 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:00.876418 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:00.881104 (+ 4686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:25:00.881109 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:25:00.883124 (+ 2015us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":62,"threads_started":1}
W20260618 23:25:00.883304 17539 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:25:00.873876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:39479 (local address 127.8.39.190:43111)
0618 23:25:00.874005 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:00.874007 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:00.874023 (+ 16us) server_negotiation.cc:407] Connection header received
0618 23:25:00.874066 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:00.874067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:00.874079 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:00.874237 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:00.874636 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:00.875124 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:00.876102 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:00.876300 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:00.876365 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:00.883218 (+ 6853us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.130:39479: BlockingRecv error: recv got EOF from 127.8.39.130:39479 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Jun 18 23:25:00 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:25:01 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:25:01.001979 17541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:00.997101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.131:40079)
0618 23:25:00.997293 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:00.997303 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:00.997359 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:00.997533 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:00.997534 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:00.997710 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:25:00.997782 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:00.997784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:00.998505 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:00.998506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:00.999034 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:00.999036 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:00.999382 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:00.999383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:00.999421 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:00.999937 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:25:00.999942 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:25:01.001882 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":115,"threads_started":1}
W20260618 23:25:01.002033 17539 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:25:00.997292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:40079 (local address 127.8.39.190:43111)
0618 23:25:00.997338 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:00.997340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:00.997350 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:25:00.997424 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:00.997424 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:00.997435 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:00.997476 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:00.997897 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:00.998412 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:00.999121 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:00.999315 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:00.999371 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:01.001965 (+ 2594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.131:40079: BlockingRecv error: recv got EOF from 127.8.39.131:40079 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Jun 18 23:25:01 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:25:01 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:25:01.758458 17542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:01.753577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.129:33971)
0618 23:25:01.753715 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:01.753726 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:01.753780 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:01.753927 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:01.753928 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:01.754093 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:25:01.754139 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:01.754140 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:01.754902 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:01.754903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:01.755492 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:01.755494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:01.755877 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:01.755878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:01.755918 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:01.756354 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:25:01.756359 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:25:01.758400 (+ 2041us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
W20260618 23:25:01.758654 17543 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:25:01.753677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:33971 (local address 127.8.39.190:43111)
0618 23:25:01.753804 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:01.753806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:01.753814 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:25:01.753837 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:01.753840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:01.753855 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:01.753913 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:01.754223 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:01.754813 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:01.755573 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:01.755811 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:01.755875 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:01.758521 (+ 2646us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.129:33971: BlockingRecv error: recv got EOF from 127.8.39.129:33971 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":81,"threads_started":1}
Jun 18 23:25:01 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:25:01 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:25:01.890014 17544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:01.883912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.130:45991)
0618 23:25:01.884381 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:01.884393 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:01.884757 (+ 364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:01.884940 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:01.884941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:01.885200 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:25:01.885263 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:01.885265 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:01.886108 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:01.886109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:01.886841 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:01.886843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:01.887196 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:01.887197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:01.887235 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:01.887932 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:25:01.887938 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:25:01.889942 (+ 2004us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":352,"threads_started":1}
W20260618 23:25:01.890156 17543 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:25:01.884190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:45991 (local address 127.8.39.190:43111)
0618 23:25:01.884359 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:01.884361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:01.884731 (+ 370us) server_negotiation.cc:407] Connection header received
0618 23:25:01.884844 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:01.884845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:01.884859 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:01.884933 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:01.885373 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:01.886015 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:01.886939 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:01.887125 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:01.887198 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:01.890060 (+ 2862us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.130:45991: BlockingRecv error: recv got EOF from 127.8.39.130:45991 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
Jun 18 23:25:02 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:25:02 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:25:02.007099 17545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:02.002655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.131:54003)
0618 23:25:02.002814 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:02.002824 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:02.002868 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:02.003014 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:02.003015 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:02.003172 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:25:02.003219 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:02.003220 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:02.003887 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:02.003888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:02.004416 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:02.004417 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:02.004788 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:02.004789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:02.004831 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:02.005269 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:25:02.005274 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:25:02.007032 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
W20260618 23:25:02.007251 17543 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:25:02.002790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:54003 (local address 127.8.39.190:43111)
0618 23:25:02.002872 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:02.002874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:02.002918 (+ 44us) server_negotiation.cc:407] Connection header received
0618 23:25:02.002939 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:02.002939 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:02.002951 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:02.002995 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:02.003286 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:02.003791 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:02.004529 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:02.004709 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:02.004771 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:02.007153 (+ 2382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.131:54003: BlockingRecv error: recv got EOF from 127.8.39.131:54003 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Jun 18 23:25:02 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:25:02 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:25:02.763686 17546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:02.759038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.129:50499)
0618 23:25:02.759192 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:02.759211 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:02.759259 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:02.759414 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:02.759415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:02.759582 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:25:02.759626 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:02.759627 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:02.760322 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:02.760323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:02.760880 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:02.760882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:02.761251 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:02.761251 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:02.761294 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:02.761766 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:25:02.761771 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:25:02.763617 (+ 1846us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":116,"threads_started":1}
W20260618 23:25:02.763801 17547 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:25:02.759110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:50499 (local address 127.8.39.190:43111)
0618 23:25:02.759253 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:02.759255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:02.759265 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:25:02.759307 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:02.759308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:02.759320 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:02.759390 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:02.759732 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:02.760234 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:02.760959 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:02.761182 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:02.761238 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:02.763721 (+ 2483us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.129:50499: BlockingRecv error: recv got EOF from 127.8.39.129:50499 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":82,"threads_started":1}
Jun 18 23:25:02 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:25:02 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:25:02.896062 17548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:02.890829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.130:35449)
0618 23:25:02.891087 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:02.891105 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:02.891312 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:02.891584 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:02.891585 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:02.891775 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:25:02.891824 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:02.891825 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:02.892608 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:02.892610 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:02.893318 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:02.893320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:02.893711 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:02.893712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:02.893765 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:02.894246 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:25:02.894251 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:25:02.895955 (+ 1704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":152,"threads_started":1}
W20260618 23:25:02.896106 17547 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:25:02.890929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:35449 (local address 127.8.39.190:43111)
0618 23:25:02.891025 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:02.891028 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:02.891275 (+ 247us) server_negotiation.cc:407] Connection header received
0618 23:25:02.891407 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:02.891409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:02.891424 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:02.891561 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:02.891909 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:02.892530 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:02.893417 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:02.893624 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:02.893711 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:02.896034 (+ 2323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.130:35449: BlockingRecv error: recv got EOF from 127.8.39.130:35449 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 18 23:25:03 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:25:03 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:25:03.012640 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:03.007658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.131:59657)
0618 23:25:03.007875 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:03.007888 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:03.007982 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:03.008144 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:03.008144 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:03.008327 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:25:03.008376 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:03.008378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:03.009222 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:03.009223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:03.009783 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:03.009785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:03.010165 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:03.010166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:03.010226 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:03.010710 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:25:03.010715 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:25:03.012573 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":102,"threads_started":1}
W20260618 23:25:03.012748 17547 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:25:03.007758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:59657 (local address 127.8.39.190:43111)
0618 23:25:03.007856 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:03.007858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:03.007965 (+ 107us) server_negotiation.cc:407] Connection header received
0618 23:25:03.008040 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:03.008041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:03.008069 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:03.008137 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:03.008456 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:03.009063 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:03.009872 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:03.010062 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:03.010167 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:03.012673 (+ 2506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.131:59657: BlockingRecv error: recv got EOF from 127.8.39.131:59657 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Jun 18 23:25:03 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:25:03 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:25:03.770817 17550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:03.764286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.129:58113)
0618 23:25:03.764788 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:03.764800 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:03.764859 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:03.765316 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:03.765318 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:03.765508 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:25:03.765559 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:03.765562 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:03.766336 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:03.766337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:03.767173 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:03.767179 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:03.767810 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:03.767812 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:03.767900 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:03.768543 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:25:03.768550 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:25:03.770742 (+ 2192us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":115,"threads_started":1}
W20260618 23:25:03.771025 17551 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:25:03.764432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:58113 (local address 127.8.39.190:43111)
0618 23:25:03.764961 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:03.764964 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:03.764981 (+ 17us) server_negotiation.cc:407] Connection header received
0618 23:25:03.765017 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:03.765018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:03.765034 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:03.765179 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:03.765692 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:03.766243 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:03.767359 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:03.767710 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:03.767792 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:03.770925 (+ 3133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.129:58113: BlockingRecv error: recv got EOF from 127.8.39.129:58113 (error 108)
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":434,"threads_started":1}
Jun 18 23:25:03 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:25:03 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:25:03.901731 17552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:03.896755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.130:41369)
0618 23:25:03.896930 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:03.896944 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:03.897034 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:03.897211 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:03.897212 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:03.897404 (+ 192us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:25:03.897459 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:03.897460 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:03.898140 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:03.898141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:03.898681 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:03.898682 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:03.899144 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:03.899145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:03.899219 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:03.899827 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:25:03.899832 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:25:03.901672 (+ 1840us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":79,"threads_started":1}
W20260618 23:25:03.901866 17551 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:25:03.896890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:41369 (local address 127.8.39.190:43111)
0618 23:25:03.896963 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:03.896965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:03.897035 (+ 70us) server_negotiation.cc:407] Connection header received
0618 23:25:03.897082 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:03.897083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:03.897095 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:03.897156 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:03.897551 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:03.898052 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:03.898770 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:03.898999 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:03.899055 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:03.901785 (+ 2730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.130:41369: BlockingRecv error: recv got EOF from 127.8.39.130:41369 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 18 23:25:04 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:25:04 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:25:04.017773 17553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:04.013171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.131:42955)
0618 23:25:04.013326 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:04.013338 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:04.013406 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:04.013540 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:04.013541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:04.013701 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:25:04.013744 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:04.013746 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:04.014491 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:04.014492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:04.015022 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:04.015024 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:04.015504 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:04.015505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:04.015576 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:04.016002 (+ 426us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:25:04.016007 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:25:04.017715 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":103,"threads_started":1}
W20260618 23:25:04.018169 17551 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:25:04.013287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:42955 (local address 127.8.39.190:43111)
0618 23:25:04.013357 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:04.013359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:04.013391 (+ 32us) server_negotiation.cc:407] Connection header received
0618 23:25:04.013464 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:04.013465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:04.013476 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:04.013530 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:04.013831 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:04.014389 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:04.015155 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:04.015352 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:04.015427 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:04.018123 (+ 2696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.131:42955: BlockingRecv error: recv got EOF from 127.8.39.131:42955 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 18 23:25:04 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:25:04 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:25:04.776747 17554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:04.771395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.129:47741)
0618 23:25:04.771536 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:04.771555 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:04.771604 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:04.771874 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:04.771877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:04.772280 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:25:04.772362 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:04.772365 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:04.773357 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:04.773358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:04.773928 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:04.773929 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:04.774370 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:04.774371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:04.774442 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:04.774891 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:25:04.774895 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:25:04.776677 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":91,"threads_started":1}
W20260618 23:25:04.776937 17555 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:25:04.771509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:47741 (local address 127.8.39.190:43111)
0618 23:25:04.771635 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:04.771636 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:04.771644 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:25:04.771666 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:04.771667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:04.771677 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:04.771744 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:04.772479 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:04.773239 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:04.774047 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:04.774242 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:04.774372 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:04.776838 (+ 2466us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.129:47741: BlockingRecv error: recv got EOF from 127.8.39.129:47741 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":65,"threads_started":1}
Jun 18 23:25:04 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:25:04 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:25:04.908278 17556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:04.902279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.130:50171)
0618 23:25:04.902424 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:04.902439 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:04.902505 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:04.902658 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:04.902659 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:04.902845 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:25:04.902889 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:04.902891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:04.903910 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:04.903911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:04.904567 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:04.904569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:04.904982 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:04.904983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:04.905047 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:04.906355 (+ 1308us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:25:04.906363 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:25:04.908182 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":99,"threads_started":1}
W20260618 23:25:04.908321 17555 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:25:04.902423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:50171 (local address 127.8.39.190:43111)
0618 23:25:04.902498 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:04.902500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:04.902510 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:25:04.902570 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:04.902571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:04.902582 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:04.902655 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:04.902972 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:04.903796 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:04.904672 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:04.904893 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:04.904980 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:04.908265 (+ 3285us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.130:50171: BlockingRecv error: recv got EOF from 127.8.39.130:50171 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 18 23:25:05 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:25:05 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:25:05.022986 17557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:05.018203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.131:47323)
0618 23:25:05.018336 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:05.018346 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:05.018420 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:05.018539 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:05.018540 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:05.018691 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:25:05.018734 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:05.018736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:05.019486 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:05.019487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:05.020049 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:05.020051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:05.020460 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:05.020460 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:05.020502 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:05.020988 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:25:05.020993 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:25:05.022920 (+ 1927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":57,"threads_started":1}
W20260618 23:25:05.023164 17555 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:25:05.018263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:47323 (local address 127.8.39.190:43111)
0618 23:25:05.018345 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:05.018347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:05.018405 (+ 58us) server_negotiation.cc:407] Connection header received
0618 23:25:05.018470 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:05.018471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:05.018482 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:05.018522 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:05.018855 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:05.019387 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:05.020155 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:05.020382 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:05.020444 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:05.023071 (+ 2627us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.131:47323: BlockingRecv error: recv got EOF from 127.8.39.131:47323 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 18 23:25:05 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:25:05 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:25:05.781760 17558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:05.777167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.129:48129)
0618 23:25:05.777350 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:05.777363 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:05.777431 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:05.777649 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:05.777650 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:05.777825 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:25:05.777870 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:05.777871 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:05.778540 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:05.778541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:05.779073 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:05.779075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:05.779389 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:05.779390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:05.779429 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:05.779973 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:25:05.779978 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:25:05.781702 (+ 1724us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":77,"threads_started":1}
W20260618 23:25:05.781913 17559 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:25:05.777384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:48129 (local address 127.8.39.190:43111)
0618 23:25:05.777519 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:05.777521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:05.777529 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:25:05.777559 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:05.777559 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:05.777569 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:05.777626 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:05.777951 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:05.778451 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:05.779150 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:05.779327 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:05.779375 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:05.781827 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.129:48129: BlockingRecv error: recv got EOF from 127.8.39.129:48129 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
Jun 18 23:25:05 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:25:05 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:25:05.920194 17560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:05.908861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.130:37657)
0618 23:25:05.909037 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:05.909049 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:05.909159 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:05.909318 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:05.909319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:05.909504 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:25:05.909548 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:05.909550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:05.910214 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:05.910214 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:05.910749 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:05.910751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:05.911137 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:05.911137 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:05.911343 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:05.917996 (+ 6653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:25:05.918001 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:25:05.920071 (+ 2070us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":68,"threads_started":1}
W20260618 23:25:05.920234 17559 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:25:05.908918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:37657 (local address 127.8.39.190:43111)
0618 23:25:05.908993 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:05.908995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:05.909137 (+ 142us) server_negotiation.cc:407] Connection header received
0618 23:25:05.909221 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:05.909222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:05.909235 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:05.909312 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:05.909635 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:05.910132 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:05.910867 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:05.911054 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:05.911137 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:05.920171 (+ 9034us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.130:37657: BlockingRecv error: recv got EOF from 127.8.39.130:37657 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Jun 18 23:25:06 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:25:06 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.131@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:25:06.028685 17561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:06.023689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.131:43155)
0618 23:25:06.023985 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:06.024002 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:06.024089 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:06.024285 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:06.024287 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:06.024456 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:25:06.024507 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:06.024508 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:06.025202 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:06.025203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:06.025723 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:06.025724 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:06.026078 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:06.026079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:06.026141 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:06.026674 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:25:06.026679 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:25:06.028576 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":209,"threads_started":1}
W20260618 23:25:06.028715 17559 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:25:06.023942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:43155 (local address 127.8.39.190:43111)
0618 23:25:06.024029 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:06.024031 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:06.024066 (+ 35us) server_negotiation.cc:407] Connection header received
0618 23:25:06.024181 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:06.024182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:06.024196 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:06.024265 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:06.024582 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:06.025069 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:06.025808 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:06.025995 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:06.026078 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:06.028651 (+ 2573us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.131:43155: BlockingRecv error: recv got EOF from 127.8.39.131:43155 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Jun 18 23:25:06 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:25:06 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.129@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:25:06.789245 17562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:06.782307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.129:33089)
0618 23:25:06.782473 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:06.782491 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:06.782563 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:06.782742 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:06.782743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:06.782898 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:25:06.782954 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:06.782956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:06.784008 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:06.784009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:06.784767 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:06.784769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:06.785585 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:06.785586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:06.785646 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:06.786668 (+ 1022us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:25:06.786674 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:25:06.789007 (+ 2333us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":67,"threads_started":1}
W20260618 23:25:06.789564 17563 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:25:06.782414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:33089 (local address 127.8.39.190:43111)
0618 23:25:06.782572 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:06.782574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:06.782582 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:25:06.782649 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:06.782649 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:06.782659 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:06.782722 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:06.783019 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:06.783768 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:06.785263 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:06.785486 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:06.785554 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:06.789516 (+ 3962us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.129:33089: BlockingRecv error: recv got EOF from 127.8.39.129:33089 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
Jun 18 23:25:06 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
Jun 18 23:25:06 dist-test-slave-47qt krb5kdc[16505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.39.130@KRBTEST.COM for oryx/127.8.39.190@KRBTEST.COM, Server not found in Kerberos database
I20260618 23:25:06.926503 17564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:06.920750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:43111 (local address 127.8.39.130:54941)
0618 23:25:06.920913 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:06.920923 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:06.920986 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:06.921185 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:06.921186 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:06.921354 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:25:06.921401 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:06.921403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:06.922089 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:06.922090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:06.922742 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:06.922744 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:06.923357 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:06.923358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:06.923451 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:06.924012 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:25:06.924017 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:25:06.926398 (+ 2381us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.39.190:43111: Server oryx/127.8.39.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":87,"threads_started":1}
W20260618 23:25:06.926529 17563 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:25:06.920868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:54941 (local address 127.8.39.190:43111)
0618 23:25:06.920912 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:06.920914 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:06.920982 (+ 68us) server_negotiation.cc:407] Connection header received
0618 23:25:06.921058 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:06.921059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:06.921069 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:06.921129 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:06.921484 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:06.921998 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:06.922880 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:06.923237 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:06.923332 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:06.926473 (+ 3141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.39.130:54941: BlockingRecv error: recv got EOF from 127.8.39.130:54941 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260618 23:25:06.938859 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 17088
I20260618 23:25:06.943305 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 17240
I20260618 23:25:06.947882 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 17372
I20260618 23:25:06.953058 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 17018
2026-06-18T23:25:06Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17490 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-18T23:25:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:25:06Z Disabled control of system clock
I20260618 23:25:06.980556 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:36733
--webserver_interface=127.8.39.190
--webserver_port=0
--builtin_ntp_servers=127.8.39.148:44607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:36733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260618 23:25:07.064261 17568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:25:07.064455 17568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:25:07.064599 17568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:25:07.066995 17568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:25:07.067162 17568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:25:07.067183 17568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:25:07.067198 17568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:25:07.067210 17568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:25:07.070525 17568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:44607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:36733
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:36733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:25:07.070770 17568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:25:07.071094 17568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:25:07.076465 17573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:25:07.076555 17576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:25:07.076431 17568 server_base.cc:1034] running on GCE node
W20260618 23:25:07.076400 17574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:25:07.076939 17568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:25:07.077283 17568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:25:07.078516 17568 hybrid_clock.cc:648] HybridClock initialized: now 1781825107078493 us; error 47 us; skew 500 ppm
I20260618 23:25:07.080020 17568 webserver.cc:460] Webserver started at http://127.8.39.190:44935/ using document root <none> and password file <none>
I20260618 23:25:07.080291 17568 fs_manager.cc:359] Metadata directory not provided
I20260618 23:25:07.080343 17568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:25:07.080467 17568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:25:07.081663 17568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "dbdd6b5e5ebc47a980373a9b2c018a22"
format_stamp: "Formatted at 2026-06-18 23:25:07 on dist-test-slave-47qt"
server_key: "04aadb15920a0eac07f57262d6c07b61"
server_key_iv: "5475121364acf6c6dc65c747a6074fd9"
server_key_version: "encryptionkey@0"
I20260618 23:25:07.082027 17568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "dbdd6b5e5ebc47a980373a9b2c018a22"
format_stamp: "Formatted at 2026-06-18 23:25:07 on dist-test-slave-47qt"
server_key: "04aadb15920a0eac07f57262d6c07b61"
server_key_iv: "5475121364acf6c6dc65c747a6074fd9"
server_key_version: "encryptionkey@0"
I20260618 23:25:07.083483 17568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:25:07.084501 17582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:25:07.084733 17568 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:25:07.084803 17568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "dbdd6b5e5ebc47a980373a9b2c018a22"
format_stamp: "Formatted at 2026-06-18 23:25:07 on dist-test-slave-47qt"
server_key: "04aadb15920a0eac07f57262d6c07b61"
server_key_iv: "5475121364acf6c6dc65c747a6074fd9"
server_key_version: "encryptionkey@0"
I20260618 23:25:07.084898 17568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:25:07.107999 17568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:25:07.108856 17568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:25:07.109211 17568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:25:07.113749 17568 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:36733
I20260618 23:25:07.113775 17634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:36733 every 8 connection(s)
I20260618 23:25:07.114122 17568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:25:07.115077 17635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:25:07.115082 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 17568
I20260618 23:25:07.115185 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:25:07.115386 8350 external_mini_cluster.cc:1428] Setting key 2e80f13fb82024862ddf5848fcea514b
I20260618 23:25:07.118430 17635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbdd6b5e5ebc47a980373a9b2c018a22: Bootstrap starting.
I20260618 23:25:07.119118 17635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dbdd6b5e5ebc47a980373a9b2c018a22: Neither blocks nor log segments found. Creating new log.
I20260618 23:25:07.119424 17635 log.cc:824] T 00000000000000000000000000000000 P dbdd6b5e5ebc47a980373a9b2c018a22: Log is configured to *not* fsync() on all Append() calls
I20260618 23:25:07.120123 17635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dbdd6b5e5ebc47a980373a9b2c018a22: No bootstrap required, opened a new log
I20260618 23:25:07.121035 17638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:07.115907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60732 (local address 127.8.39.190:36733)
0618 23:25:07.116120 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:07.116127 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:07.116150 (+ 23us) server_negotiation.cc:407] Connection header received
0618 23:25:07.116608 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:07.116615 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:07.116671 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:07.116780 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:07.117032 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:07.117952 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:07.119022 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:07.119466 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:07.119627 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:07.120331 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:25:07.120334 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:25:07.120335 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:25:07.120527 (+ 192us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:25:07.120566 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:25:07.120579 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:25:07.120649 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0618 23:25:07.120736 (+ 87us) server_negotiation.cc:299] Negotiation successful
0618 23:25:07.120809 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":97,"threads_started":1}
I20260618 23:25:07.122370 17635 raft_consensus.cc:348] T 00000000000000000000000000000000 P dbdd6b5e5ebc47a980373a9b2c018a22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbdd6b5e5ebc47a980373a9b2c018a22" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 36733 } }
I20260618 23:25:07.122504 17635 raft_consensus.cc:374] T 00000000000000000000000000000000 P dbdd6b5e5ebc47a980373a9b2c018a22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:25:07.122526 17635 raft_consensus.cc:729] T 00000000000000000000000000000000 P dbdd6b5e5ebc47a980373a9b2c018a22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbdd6b5e5ebc47a980373a9b2c018a22, State: Initialized, Role: FOLLOWER
I20260618 23:25:07.122654 17635 consensus_queue.cc:260] T 00000000000000000000000000000000 P dbdd6b5e5ebc47a980373a9b2c018a22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbdd6b5e5ebc47a980373a9b2c018a22" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 36733 } }
I20260618 23:25:07.122728 17635 raft_consensus.cc:388] T 00000000000000000000000000000000 P dbdd6b5e5ebc47a980373a9b2c018a22 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:25:07.122750 17635 raft_consensus.cc:482] T 00000000000000000000000000000000 P dbdd6b5e5ebc47a980373a9b2c018a22 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:25:07.122798 17635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dbdd6b5e5ebc47a980373a9b2c018a22 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:25:07.123299 17635 raft_consensus.cc:504] T 00000000000000000000000000000000 P dbdd6b5e5ebc47a980373a9b2c018a22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbdd6b5e5ebc47a980373a9b2c018a22" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 36733 } }
I20260618 23:25:07.123378 17635 leader_election.cc:302] T 00000000000000000000000000000000 P dbdd6b5e5ebc47a980373a9b2c018a22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dbdd6b5e5ebc47a980373a9b2c018a22; no voters:
I20260618 23:25:07.123495 17635 leader_election.cc:288] T 00000000000000000000000000000000 P dbdd6b5e5ebc47a980373a9b2c018a22 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:25:07.123705 17640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dbdd6b5e5ebc47a980373a9b2c018a22 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:25:07.123785 17635 sys_catalog.cc:564] T 00000000000000000000000000000000 P dbdd6b5e5ebc47a980373a9b2c018a22 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:25:07.123975 17640 raft_consensus.cc:686] T 00000000000000000000000000000000 P dbdd6b5e5ebc47a980373a9b2c018a22 [term 1 LEADER]: Becoming Leader. State: Replica: dbdd6b5e5ebc47a980373a9b2c018a22, State: Running, Role: LEADER
I20260618 23:25:07.124368 17640 consensus_queue.cc:237] T 00000000000000000000000000000000 P dbdd6b5e5ebc47a980373a9b2c018a22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbdd6b5e5ebc47a980373a9b2c018a22" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 36733 } }
I20260618 23:25:07.125308 17642 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbdd6b5e5ebc47a980373a9b2c018a22 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dbdd6b5e5ebc47a980373a9b2c018a22. Latest consensus state: current_term: 1 leader_uuid: "dbdd6b5e5ebc47a980373a9b2c018a22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbdd6b5e5ebc47a980373a9b2c018a22" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 36733 } } }
I20260618 23:25:07.125442 17642 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbdd6b5e5ebc47a980373a9b2c018a22 [sys.catalog]: This master's current role is: LEADER
I20260618 23:25:07.125450 17641 sys_catalog.cc:455] T 00000000000000000000000000000000 P dbdd6b5e5ebc47a980373a9b2c018a22 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dbdd6b5e5ebc47a980373a9b2c018a22" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dbdd6b5e5ebc47a980373a9b2c018a22" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 36733 } } }
I20260618 23:25:07.125533 17641 sys_catalog.cc:458] T 00000000000000000000000000000000 P dbdd6b5e5ebc47a980373a9b2c018a22 [sys.catalog]: This master's current role is: LEADER
W20260618 23:25:07.125686 17654 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dbdd6b5e5ebc47a980373a9b2c018a22: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 23:25:07.125731 17654 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:25:07.125877 17656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:25:07.126029 17656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:25:07.127385 17656 catalog_manager.cc:1347] Generated new cluster ID: 6ec0d008fe5a4743854c9d7c0315b7e9
I20260618 23:25:07.127446 17656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:25:07.132586 17656 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:25:07.133459 17656 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:25:07.136745 17656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dbdd6b5e5ebc47a980373a9b2c018a22: Generated new TSK 0
I20260618 23:25:07.136901 17656 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:25:07.140539 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.129:0
--local_ip_for_outbound_sockets=127.8.39.129
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:36733
--builtin_ntp_servers=127.8.39.148:44607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260618 23:25:07.227869 17659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:25:07.228323 17659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:25:07.228374 17659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:25:07.230238 17659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:25:07.230343 17659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:25:07.230404 17659 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.129
I20260618 23:25:07.232825 17659 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:44607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:36733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17659
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.129
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:25:07.233240 17659 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:25:07.233688 17659 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:25:07.239418 17664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:25:07.239666 17659 server_base.cc:1034] running on GCE node
W20260618 23:25:07.239797 17667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:25:07.239799 17665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:25:07.240149 17659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:25:07.240439 17659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:25:07.241603 17659 hybrid_clock.cc:648] HybridClock initialized: now 1781825107241592 us; error 37 us; skew 500 ppm
I20260618 23:25:07.242970 17659 webserver.cc:460] Webserver started at http://127.8.39.129:44377/ using document root <none> and password file <none>
I20260618 23:25:07.243227 17659 fs_manager.cc:359] Metadata directory not provided
I20260618 23:25:07.243292 17659 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:25:07.243423 17659 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:25:07.244782 17659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/data/instance:
uuid: "8714357d42594fd2899db47efa151727"
format_stamp: "Formatted at 2026-06-18 23:25:07 on dist-test-slave-47qt"
server_key: "1ce15bb7559dcbe09b25702e9757ff44"
server_key_iv: "3bf6287ca479f8a322ee89a06be09186"
server_key_version: "encryptionkey@0"
I20260618 23:25:07.245208 17659 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance:
uuid: "8714357d42594fd2899db47efa151727"
format_stamp: "Formatted at 2026-06-18 23:25:07 on dist-test-slave-47qt"
server_key: "1ce15bb7559dcbe09b25702e9757ff44"
server_key_iv: "3bf6287ca479f8a322ee89a06be09186"
server_key_version: "encryptionkey@0"
I20260618 23:25:07.246940 17659 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260618 23:25:07.247845 17673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:25:07.248059 17659 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:25:07.248153 17659 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/wal
uuid: "8714357d42594fd2899db47efa151727"
format_stamp: "Formatted at 2026-06-18 23:25:07 on dist-test-slave-47qt"
server_key: "1ce15bb7559dcbe09b25702e9757ff44"
server_key_iv: "3bf6287ca479f8a322ee89a06be09186"
server_key_version: "encryptionkey@0"
I20260618 23:25:07.248219 17659 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:25:07.265372 17659 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:25:07.265702 17659 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:25:07.265832 17659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:25:07.266049 17659 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:25:07.266360 17659 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:25:07.266396 17659 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:25:07.266431 17659 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:25:07.266456 17659 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:25:07.273949 17659 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.129:37421
I20260618 23:25:07.273978 17786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.129:37421 every 8 connection(s)
I20260618 23:25:07.274329 17659 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
I20260618 23:25:07.276722 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 17659
I20260618 23:25:07.276830 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance
I20260618 23:25:07.277006 8350 external_mini_cluster.cc:1428] Setting key 36cb719d7fb7e1cab10f5a04bd7dd56e
I20260618 23:25:07.278028 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.130:0
--local_ip_for_outbound_sockets=127.8.39.130
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:36733
--builtin_ntp_servers=127.8.39.148:44607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260618 23:25:07.280316 17638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:07.275040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:57359 (local address 127.8.39.190:36733)
0618 23:25:07.275106 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:07.275109 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:07.275349 (+ 240us) server_negotiation.cc:407] Connection header received
0618 23:25:07.275845 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:07.275846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:07.275860 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:07.275939 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:07.276528 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:07.277298 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:07.278719 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:07.278919 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:07.279021 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:07.279998 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:25:07.280000 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:25:07.280000 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:25:07.280005 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:25:07.280020 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:25:07.280025 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:25:07.280089 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0618 23:25:07.280225 (+ 136us) server_negotiation.cc:299] Negotiation successful
0618 23:25:07.280251 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260618 23:25:07.280575 17789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:07.275037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:36733 (local address 127.8.39.129:57359)
0618 23:25:07.275221 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:07.275260 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:07.275754 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:07.275987 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:07.275990 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:07.275998 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:25:07.276444 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:07.276447 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:07.277382 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:07.277383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:07.278612 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:07.278615 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:07.279037 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:07.279038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:07.279217 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:07.279844 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:25:07.279845 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:25:07.279872 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:25:07.279893 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:25:07.280122 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:25:07.280127 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:25:07.280130 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:25:07.280200 (+ 70us) client_negotiation.cc:240] Negotiation successful
0618 23:25:07.280305 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":96,"threads_started":1}
I20260618 23:25:07.281082 17787 heartbeater.cc:347] Connected to a master server at 127.8.39.190:36733
I20260618 23:25:07.281208 17787 heartbeater.cc:464] Registering TS with master...
I20260618 23:25:07.281381 17787 heartbeater.cc:511] Master 127.8.39.190:36733 requested a full tablet report, sending...
I20260618 23:25:07.281903 17599 ts_manager.cc:194] Registered new tserver with Master: 8714357d42594fd2899db47efa151727 (127.8.39.129:37421)
I20260618 23:25:07.282502 17599 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.39.129:57359
W20260618 23:25:07.359501 17790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:25:07.359685 17790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:25:07.359706 17790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:25:07.360836 17790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:25:07.360875 17790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:25:07.360900 17790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.130
I20260618 23:25:07.362212 17790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:44607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:36733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.130
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:25:07.362438 17790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:25:07.362752 17790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:25:07.368207 17796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:25:07.368206 17798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:25:07.368355 17795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:25:07.368687 17790 server_base.cc:1034] running on GCE node
I20260618 23:25:07.368917 17790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:25:07.369371 17790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:25:07.370556 17790 hybrid_clock.cc:648] HybridClock initialized: now 1781825107370540 us; error 28 us; skew 500 ppm
I20260618 23:25:07.371927 17790 webserver.cc:460] Webserver started at http://127.8.39.130:42893/ using document root <none> and password file <none>
I20260618 23:25:07.372193 17790 fs_manager.cc:359] Metadata directory not provided
I20260618 23:25:07.372263 17790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:25:07.372414 17790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:25:07.373576 17790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/data/instance:
uuid: "074548730e144082b7b849132e30841b"
format_stamp: "Formatted at 2026-06-18 23:25:07 on dist-test-slave-47qt"
server_key: "ca4579dae6747d733d172bd923a328c5"
server_key_iv: "d34f1632f1f65a7df66a30c6b69e8788"
server_key_version: "encryptionkey@0"
I20260618 23:25:07.373919 17790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance:
uuid: "074548730e144082b7b849132e30841b"
format_stamp: "Formatted at 2026-06-18 23:25:07 on dist-test-slave-47qt"
server_key: "ca4579dae6747d733d172bd923a328c5"
server_key_iv: "d34f1632f1f65a7df66a30c6b69e8788"
server_key_version: "encryptionkey@0"
I20260618 23:25:07.375170 17790 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:25:07.375875 17804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:25:07.376092 17790 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260618 23:25:07.376189 17790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/wal
uuid: "074548730e144082b7b849132e30841b"
format_stamp: "Formatted at 2026-06-18 23:25:07 on dist-test-slave-47qt"
server_key: "ca4579dae6747d733d172bd923a328c5"
server_key_iv: "d34f1632f1f65a7df66a30c6b69e8788"
server_key_version: "encryptionkey@0"
I20260618 23:25:07.376271 17790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:25:07.400535 17790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:25:07.400857 17790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:25:07.401012 17790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:25:07.401289 17790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:25:07.401625 17790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:25:07.401680 17790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:25:07.401717 17790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:25:07.401744 17790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:25:07.410173 17790 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.130:45177
I20260618 23:25:07.410293 17917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.130:45177 every 8 connection(s)
I20260618 23:25:07.410739 17790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
I20260618 23:25:07.413318 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 17790
I20260618 23:25:07.413425 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance
I20260618 23:25:07.413607 8350 external_mini_cluster.cc:1428] Setting key e06f53f0cc5e5759173d01f3098902ef
I20260618 23:25:07.414678 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.131:0
--local_ip_for_outbound_sockets=127.8.39.131
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:36733
--builtin_ntp_servers=127.8.39.148:44607
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260618 23:25:07.417196 17638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:07.411744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:51221 (local address 127.8.39.190:36733)
0618 23:25:07.411818 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:07.411821 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:07.412067 (+ 246us) server_negotiation.cc:407] Connection header received
0618 23:25:07.412768 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:07.412771 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:07.412786 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:07.412844 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:07.413640 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:07.414253 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:07.415759 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:07.416020 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:07.416098 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:07.416921 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:25:07.416923 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:25:07.416923 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:25:07.416929 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:25:07.416940 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:25:07.416945 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:25:07.417000 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0618 23:25:07.417068 (+ 68us) server_negotiation.cc:299] Negotiation successful
0618 23:25:07.417092 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260618 23:25:07.417433 17920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:07.411665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:36733 (local address 127.8.39.130:51221)
0618 23:25:07.411919 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:07.411959 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:07.412587 (+ 628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:07.412921 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:07.412925 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:07.412935 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:25:07.413550 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:07.413554 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:07.414352 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:07.414354 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:07.415670 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:07.415673 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:07.416116 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:07.416117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:07.416223 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:07.416784 (+ 561us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:25:07.416784 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:25:07.416811 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:25:07.416831 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:25:07.417002 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:25:07.417007 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:25:07.417009 (+ 2us) client_negotiation.cc:769] Sending connection context
0618 23:25:07.417093 (+ 84us) client_negotiation.cc:240] Negotiation successful
0618 23:25:07.417224 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":119,"threads_started":1}
I20260618 23:25:07.417897 17918 heartbeater.cc:347] Connected to a master server at 127.8.39.190:36733
I20260618 23:25:07.418007 17918 heartbeater.cc:464] Registering TS with master...
I20260618 23:25:07.418196 17918 heartbeater.cc:511] Master 127.8.39.190:36733 requested a full tablet report, sending...
I20260618 23:25:07.418574 17599 ts_manager.cc:194] Registered new tserver with Master: 074548730e144082b7b849132e30841b (127.8.39.130:45177)
I20260618 23:25:07.419106 17599 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.39.130:51221
W20260618 23:25:07.499037 17921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:25:07.499228 17921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:25:07.499248 17921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:25:07.500350 17921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:25:07.500391 17921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:25:07.500416 17921 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.131
I20260618 23:25:07.501812 17921 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:44607
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:36733
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.131
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:25:07.502055 17921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:25:07.502350 17921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:25:07.508819 17926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:25:07.508816 17929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:25:07.508816 17927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:25:07.509233 17921 server_base.cc:1034] running on GCE node
I20260618 23:25:07.509480 17921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:25:07.509737 17921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:25:07.510926 17921 hybrid_clock.cc:648] HybridClock initialized: now 1781825107510850 us; error 93 us; skew 500 ppm
I20260618 23:25:07.512648 17921 webserver.cc:460] Webserver started at http://127.8.39.131:45317/ using document root <none> and password file <none>
I20260618 23:25:07.512913 17921 fs_manager.cc:359] Metadata directory not provided
I20260618 23:25:07.512984 17921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:25:07.513167 17921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:25:07.514232 17921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/data/instance:
uuid: "edf008a1197444bda24538f605a1fa87"
format_stamp: "Formatted at 2026-06-18 23:25:07 on dist-test-slave-47qt"
server_key: "79b129bf7cddfad42dba9adcb54ae1e2"
server_key_iv: "2a3608f4f8cbb44abb80ea525d7787cb"
server_key_version: "encryptionkey@0"
I20260618 23:25:07.514587 17921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance:
uuid: "edf008a1197444bda24538f605a1fa87"
format_stamp: "Formatted at 2026-06-18 23:25:07 on dist-test-slave-47qt"
server_key: "79b129bf7cddfad42dba9adcb54ae1e2"
server_key_iv: "2a3608f4f8cbb44abb80ea525d7787cb"
server_key_version: "encryptionkey@0"
I20260618 23:25:07.515885 17921 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:25:07.516564 17935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:25:07.516749 17921 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260618 23:25:07.516840 17921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/wal
uuid: "edf008a1197444bda24538f605a1fa87"
format_stamp: "Formatted at 2026-06-18 23:25:07 on dist-test-slave-47qt"
server_key: "79b129bf7cddfad42dba9adcb54ae1e2"
server_key_iv: "2a3608f4f8cbb44abb80ea525d7787cb"
server_key_version: "encryptionkey@0"
I20260618 23:25:07.516925 17921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:25:07.529465 17921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:25:07.529779 17921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:25:07.529925 17921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:25:07.530158 17921 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:25:07.530468 17921 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:25:07.530524 17921 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:25:07.530565 17921 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:25:07.530594 17921 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:25:07.536175 17921 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.131:35241
I20260618 23:25:07.536218 18048 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.131:35241 every 8 connection(s)
I20260618 23:25:07.536514 17921 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
I20260618 23:25:07.539985 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 17921
I20260618 23:25:07.540084 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance
I20260618 23:25:07.540261 8350 external_mini_cluster.cc:1428] Setting key 539b039556f7d0fe0790b0f69f60cbc8
I20260618 23:25:07.541652 17638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:07.537143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:46601 (local address 127.8.39.190:36733)
0618 23:25:07.537193 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:07.537195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:07.537434 (+ 239us) server_negotiation.cc:407] Connection header received
0618 23:25:07.537904 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:07.537905 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:07.537915 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:07.537964 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:07.538562 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:07.539094 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:07.540134 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:07.540404 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:07.540479 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:07.541428 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:25:07.541429 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:25:07.541429 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:25:07.541433 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:25:07.541445 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:25:07.541449 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:25:07.541496 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0618 23:25:07.541578 (+ 82us) server_negotiation.cc:299] Negotiation successful
0618 23:25:07.541598 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260618 23:25:07.541816 18051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:07.537158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:36733 (local address 127.8.39.131:46601)
0618 23:25:07.537332 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:07.537364 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:07.537819 (+ 455us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:07.538014 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:07.538017 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:07.538024 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:25:07.538482 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:07.538486 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:07.539168 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:07.539169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:07.540044 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:07.540048 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:07.540523 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:07.540524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:07.540678 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:07.541301 (+ 623us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:25:07.541301 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:25:07.541328 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:25:07.541348 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:25:07.541499 (+ 151us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:25:07.541501 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:25:07.541502 (+ 1us) client_negotiation.cc:769] Sending connection context
0618 23:25:07.541553 (+ 51us) client_negotiation.cc:240] Negotiation successful
0618 23:25:07.541634 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":81,"threads_started":1}
I20260618 23:25:07.542274 18049 heartbeater.cc:347] Connected to a master server at 127.8.39.190:36733
I20260618 23:25:07.542376 18049 heartbeater.cc:464] Registering TS with master...
I20260618 23:25:07.542544 18049 heartbeater.cc:511] Master 127.8.39.190:36733 requested a full tablet report, sending...
I20260618 23:25:07.542883 17599 ts_manager.cc:194] Registered new tserver with Master: edf008a1197444bda24538f605a1fa87 (127.8.39.131:35241)
I20260618 23:25:07.543248 17599 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.39.131:46601
I20260618 23:25:07.553068 8350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260618 23:25:07.555254 17638 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:25:07.554505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60738 (local address 127.8.39.190:36733)
0618 23:25:07.554549 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:07.554551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:07.554695 (+ 144us) server_negotiation.cc:407] Connection header received
0618 23:25:07.554715 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:07.554716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:07.554724 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:07.554772 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:07.555207 (+ 435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60738: BlockingRecv error: recv got EOF from 127.0.0.1:60738 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
I20260618 23:25:07.555402 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 17659
I20260618 23:25:07.559916 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 17790
I20260618 23:25:07.564050 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 17921
I20260618 23:25:07.568424 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 17568
2026-06-18T23:25:07Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (613 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-18T23:25:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:25:07Z Disabled control of system clock
I20260618 23:25:07.594453 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:40183
--webserver_interface=127.8.39.190
--webserver_port=0
--builtin_ntp_servers=127.8.39.148:46375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:40183
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260618 23:25:07.676299 18060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:25:07.676487 18060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:25:07.676507 18060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:25:07.677554 18060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:25:07.677587 18060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:25:07.677598 18060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:25:07.677611 18060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:25:07.677623 18060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:25:07.678849 18060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:46375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:40183
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:40183
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18060
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:25:07.679021 18060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:25:07.679234 18060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:25:07.683977 18065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:25:07.683985 18066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:25:07.684175 18068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:25:07.684239 18060 server_base.cc:1034] running on GCE node
I20260618 23:25:07.684502 18060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:25:07.684787 18060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:25:07.685976 18060 hybrid_clock.cc:648] HybridClock initialized: now 1781825107685955 us; error 37 us; skew 500 ppm
I20260618 23:25:07.687654 18060 webserver.cc:460] Webserver started at http://127.8.39.190:39667/ using document root <none> and password file <none>
I20260618 23:25:07.688073 18060 fs_manager.cc:359] Metadata directory not provided
I20260618 23:25:07.688158 18060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:25:07.688310 18060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:25:07.689396 18060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "5fd102c8f2ff45b7b248ccf8d37e6f0b"
format_stamp: "Formatted at 2026-06-18 23:25:07 on dist-test-slave-47qt"
server_key: "654290f0741aa65815db826bce944233"
server_key_iv: "f43793c2c1b513a33b012db0301fe964"
server_key_version: "encryptionkey@0"
I20260618 23:25:07.689740 18060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "5fd102c8f2ff45b7b248ccf8d37e6f0b"
format_stamp: "Formatted at 2026-06-18 23:25:07 on dist-test-slave-47qt"
server_key: "654290f0741aa65815db826bce944233"
server_key_iv: "f43793c2c1b513a33b012db0301fe964"
server_key_version: "encryptionkey@0"
I20260618 23:25:07.691027 18060 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:25:07.691798 18074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:25:07.691992 18060 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260618 23:25:07.692102 18060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "5fd102c8f2ff45b7b248ccf8d37e6f0b"
format_stamp: "Formatted at 2026-06-18 23:25:07 on dist-test-slave-47qt"
server_key: "654290f0741aa65815db826bce944233"
server_key_iv: "f43793c2c1b513a33b012db0301fe964"
server_key_version: "encryptionkey@0"
I20260618 23:25:07.692181 18060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:25:07.695632 18060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:25:07.695881 18060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:25:07.696030 18060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:25:07.700043 18060 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:40183
I20260618 23:25:07.700268 18126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:40183 every 8 connection(s)
I20260618 23:25:07.700421 18060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:25:07.701198 18127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:25:07.703763 18127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fd102c8f2ff45b7b248ccf8d37e6f0b: Bootstrap starting.
I20260618 23:25:07.704319 18127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5fd102c8f2ff45b7b248ccf8d37e6f0b: Neither blocks nor log segments found. Creating new log.
I20260618 23:25:07.704613 18127 log.cc:824] T 00000000000000000000000000000000 P 5fd102c8f2ff45b7b248ccf8d37e6f0b: Log is configured to *not* fsync() on all Append() calls
I20260618 23:25:07.705361 18127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5fd102c8f2ff45b7b248ccf8d37e6f0b: No bootstrap required, opened a new log
I20260618 23:25:07.706867 18127 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5fd102c8f2ff45b7b248ccf8d37e6f0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd102c8f2ff45b7b248ccf8d37e6f0b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 40183 } }
I20260618 23:25:07.706993 18127 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5fd102c8f2ff45b7b248ccf8d37e6f0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:25:07.707015 18127 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5fd102c8f2ff45b7b248ccf8d37e6f0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5fd102c8f2ff45b7b248ccf8d37e6f0b, State: Initialized, Role: FOLLOWER
I20260618 23:25:07.707159 18127 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5fd102c8f2ff45b7b248ccf8d37e6f0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd102c8f2ff45b7b248ccf8d37e6f0b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 40183 } }
I20260618 23:25:07.707232 18127 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5fd102c8f2ff45b7b248ccf8d37e6f0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:25:07.707274 18127 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5fd102c8f2ff45b7b248ccf8d37e6f0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:25:07.707320 18127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5fd102c8f2ff45b7b248ccf8d37e6f0b [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:25:07.707821 18127 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5fd102c8f2ff45b7b248ccf8d37e6f0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd102c8f2ff45b7b248ccf8d37e6f0b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 40183 } }
I20260618 23:25:07.707924 18127 leader_election.cc:302] T 00000000000000000000000000000000 P 5fd102c8f2ff45b7b248ccf8d37e6f0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5fd102c8f2ff45b7b248ccf8d37e6f0b; no voters:
I20260618 23:25:07.708092 18127 leader_election.cc:288] T 00000000000000000000000000000000 P 5fd102c8f2ff45b7b248ccf8d37e6f0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:25:07.708124 18130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5fd102c8f2ff45b7b248ccf8d37e6f0b [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:25:07.708348 18130 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5fd102c8f2ff45b7b248ccf8d37e6f0b [term 1 LEADER]: Becoming Leader. State: Replica: 5fd102c8f2ff45b7b248ccf8d37e6f0b, State: Running, Role: LEADER
I20260618 23:25:07.708407 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 18060
I20260618 23:25:07.708489 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:25:07.708428 18130 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5fd102c8f2ff45b7b248ccf8d37e6f0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd102c8f2ff45b7b248ccf8d37e6f0b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 40183 } }
I20260618 23:25:07.708441 18127 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5fd102c8f2ff45b7b248ccf8d37e6f0b [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:25:07.708662 8350 external_mini_cluster.cc:1428] Setting key 4f68bada5e308c723ff1a841e4be6819
I20260618 23:25:07.708779 18131 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fd102c8f2ff45b7b248ccf8d37e6f0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5fd102c8f2ff45b7b248ccf8d37e6f0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd102c8f2ff45b7b248ccf8d37e6f0b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 40183 } } }
I20260618 23:25:07.708845 18131 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fd102c8f2ff45b7b248ccf8d37e6f0b [sys.catalog]: This master's current role is: LEADER
I20260618 23:25:07.709224 18132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5fd102c8f2ff45b7b248ccf8d37e6f0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5fd102c8f2ff45b7b248ccf8d37e6f0b. Latest consensus state: current_term: 1 leader_uuid: "5fd102c8f2ff45b7b248ccf8d37e6f0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5fd102c8f2ff45b7b248ccf8d37e6f0b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 40183 } } }
I20260618 23:25:07.709292 18132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5fd102c8f2ff45b7b248ccf8d37e6f0b [sys.catalog]: This master's current role is: LEADER
I20260618 23:25:07.709740 18135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:25:07.710321 18135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:25:07.711457 18141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:07.709294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36098 (local address 127.8.39.190:40183)
0618 23:25:07.709937 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:07.709945 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:07.709975 (+ 30us) server_negotiation.cc:407] Connection header received
0618 23:25:07.710106 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:07.710114 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:07.710160 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:07.710316 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:07.710967 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:25:07.710970 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:25:07.710970 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:25:07.710975 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:25:07.711015 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:25:07.711039 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:25:07.711094 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0618 23:25:07.711211 (+ 117us) server_negotiation.cc:299] Negotiation successful
0618 23:25:07.711265 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":65,"threads_started":1}
I20260618 23:25:07.712129 18135 catalog_manager.cc:1347] Generated new cluster ID: 2a8487d8d8fb4e92b0abb4e5aff06a5f
I20260618 23:25:07.712181 18135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:25:07.727568 18135 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:25:07.727727 18135 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:25:07.734545 18135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5fd102c8f2ff45b7b248ccf8d37e6f0b: Generated new TSK 0
I20260618 23:25:07.734736 18135 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:25:07.738584 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.129:0
--local_ip_for_outbound_sockets=127.8.39.129
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:40183
--builtin_ntp_servers=127.8.39.148:46375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260618 23:25:07.823052 18151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:25:07.823290 18151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:25:07.823328 18151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:25:07.824613 18151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:25:07.824698 18151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:25:07.824749 18151 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.129
I20260618 23:25:07.826433 18151 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:46375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:40183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.129
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:25:07.826685 18151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:25:07.827261 18151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:25:07.833736 18156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:25:07.833739 18157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:25:07.833771 18159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:25:07.833736 18151 server_base.cc:1034] running on GCE node
I20260618 23:25:07.834170 18151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:25:07.834427 18151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:25:07.835623 18151 hybrid_clock.cc:648] HybridClock initialized: now 1781825107835597 us; error 41 us; skew 500 ppm
I20260618 23:25:07.837005 18151 webserver.cc:460] Webserver started at http://127.8.39.129:45337/ using document root <none> and password file <none>
I20260618 23:25:07.837294 18151 fs_manager.cc:359] Metadata directory not provided
I20260618 23:25:07.837368 18151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:25:07.837525 18151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:25:07.838608 18151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/data/instance:
uuid: "1bc9085e19a847c3b6f09d8c874b3eb3"
format_stamp: "Formatted at 2026-06-18 23:25:07 on dist-test-slave-47qt"
server_key: "34e310288a680b0862ffee3950ec9275"
server_key_iv: "59ab38463d6f11c7d30579cce3bb57de"
server_key_version: "encryptionkey@0"
I20260618 23:25:07.838981 18151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance:
uuid: "1bc9085e19a847c3b6f09d8c874b3eb3"
format_stamp: "Formatted at 2026-06-18 23:25:07 on dist-test-slave-47qt"
server_key: "34e310288a680b0862ffee3950ec9275"
server_key_iv: "59ab38463d6f11c7d30579cce3bb57de"
server_key_version: "encryptionkey@0"
I20260618 23:25:07.840348 18151 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:25:07.841627 18165 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:25:07.841952 18151 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:25:07.842125 18151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/wal
uuid: "1bc9085e19a847c3b6f09d8c874b3eb3"
format_stamp: "Formatted at 2026-06-18 23:25:07 on dist-test-slave-47qt"
server_key: "34e310288a680b0862ffee3950ec9275"
server_key_iv: "59ab38463d6f11c7d30579cce3bb57de"
server_key_version: "encryptionkey@0"
I20260618 23:25:07.842285 18151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:25:07.846307 18151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:25:07.846580 18151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:25:07.846726 18151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:25:07.846936 18151 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:25:07.847249 18151 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:25:07.847302 18151 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:25:07.847342 18151 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:25:07.847369 18151 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:25:07.853379 18151 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.129:44505
I20260618 23:25:07.853417 18278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.129:44505 every 8 connection(s)
I20260618 23:25:07.853897 18151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
I20260618 23:25:07.857714 18141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:07.854886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:33659 (local address 127.8.39.190:40183)
0618 23:25:07.855021 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:07.855023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:07.855255 (+ 232us) server_negotiation.cc:407] Connection header received
0618 23:25:07.855851 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:07.855852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:07.855863 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:07.855925 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:07.857406 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:25:07.857407 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:25:07.857408 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:25:07.857415 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:25:07.857443 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:25:07.857448 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:25:07.857522 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 23:25:07.857624 (+ 102us) server_negotiation.cc:299] Negotiation successful
0618 23:25:07.857646 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260618 23:25:07.857952 18281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:07.854837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:40183 (local address 127.8.39.129:33659)
0618 23:25:07.855100 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:07.855154 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:07.855739 (+ 585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:07.856018 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:07.856022 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:07.856031 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:25:07.857181 (+ 1150us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:25:07.857181 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:25:07.857213 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:25:07.857252 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:25:07.857520 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:25:07.857527 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:25:07.857530 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:25:07.857620 (+ 90us) client_negotiation.cc:240] Negotiation successful
0618 23:25:07.857690 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":108,"threads_started":1}
I20260618 23:25:07.858680 18279 heartbeater.cc:347] Connected to a master server at 127.8.39.190:40183
I20260618 23:25:07.858776 18279 heartbeater.cc:464] Registering TS with master...
I20260618 23:25:07.858946 18279 heartbeater.cc:511] Master 127.8.39.190:40183 requested a full tablet report, sending...
I20260618 23:25:07.859580 18091 ts_manager.cc:194] Registered new tserver with Master: 1bc9085e19a847c3b6f09d8c874b3eb3 (127.8.39.129:44505)
I20260618 23:25:07.863862 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 18151
I20260618 23:25:07.864042 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance
I20260618 23:25:07.864223 8350 external_mini_cluster.cc:1428] Setting key 1ec93a02a042212248d5c4137ac6b85f
I20260618 23:25:07.865425 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.130:0
--local_ip_for_outbound_sockets=127.8.39.130
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:40183
--builtin_ntp_servers=127.8.39.148:46375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260618 23:25:07.947647 18282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:25:07.947834 18282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:25:07.947856 18282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:25:07.948900 18282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:25:07.948935 18282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:25:07.948959 18282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.130
I20260618 23:25:07.950263 18282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:46375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:40183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18282
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.130
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:25:07.950436 18282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:25:07.950681 18282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:25:07.955736 18287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:25:07.955895 18288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:25:07.955878 18282 server_base.cc:1034] running on GCE node
W20260618 23:25:07.955950 18290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:25:07.956135 18282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:25:07.956385 18282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:25:07.957564 18282 hybrid_clock.cc:648] HybridClock initialized: now 1781825107957555 us; error 39 us; skew 500 ppm
I20260618 23:25:07.958861 18282 webserver.cc:460] Webserver started at http://127.8.39.130:41325/ using document root <none> and password file <none>
I20260618 23:25:07.959108 18282 fs_manager.cc:359] Metadata directory not provided
I20260618 23:25:07.959174 18282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:25:07.959349 18282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:25:07.960361 18282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/data/instance:
uuid: "392c0caef3804f7ba1ff8706fad710b6"
format_stamp: "Formatted at 2026-06-18 23:25:07 on dist-test-slave-47qt"
server_key: "ec4f46c989edfae182effefe0c7e49f2"
server_key_iv: "73b9839121856531f586943e27093ad7"
server_key_version: "encryptionkey@0"
I20260618 23:25:07.960708 18282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance:
uuid: "392c0caef3804f7ba1ff8706fad710b6"
format_stamp: "Formatted at 2026-06-18 23:25:07 on dist-test-slave-47qt"
server_key: "ec4f46c989edfae182effefe0c7e49f2"
server_key_iv: "73b9839121856531f586943e27093ad7"
server_key_version: "encryptionkey@0"
I20260618 23:25:07.962110 18282 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:25:07.963141 18296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:25:07.963403 18282 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:25:07.963510 18282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/wal
uuid: "392c0caef3804f7ba1ff8706fad710b6"
format_stamp: "Formatted at 2026-06-18 23:25:07 on dist-test-slave-47qt"
server_key: "ec4f46c989edfae182effefe0c7e49f2"
server_key_iv: "73b9839121856531f586943e27093ad7"
server_key_version: "encryptionkey@0"
I20260618 23:25:07.963588 18282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:25:07.966917 18282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:25:07.967175 18282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:25:07.967312 18282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:25:07.967517 18282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:25:07.967794 18282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:25:07.967844 18282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:25:07.967880 18282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:25:07.967906 18282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:25:07.974242 18282 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.130:37549
I20260618 23:25:07.974288 18409 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.130:37549 every 8 connection(s)
I20260618 23:25:07.974593 18282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
I20260618 23:25:07.977521 18141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:07.975445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:46745 (local address 127.8.39.190:40183)
0618 23:25:07.975491 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:07.975493 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:07.975602 (+ 109us) server_negotiation.cc:407] Connection header received
0618 23:25:07.976062 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:07.976063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:07.976069 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:07.976139 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:07.977323 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:25:07.977325 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:25:07.977325 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:25:07.977329 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:25:07.977343 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:25:07.977347 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:25:07.977412 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0618 23:25:07.977459 (+ 47us) server_negotiation.cc:299] Negotiation successful
0618 23:25:07.977471 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260618 23:25:07.977696 18412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:07.975309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:40183 (local address 127.8.39.130:46745)
0618 23:25:07.975480 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:07.975527 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:07.975951 (+ 424us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:07.976191 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:07.976194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:07.976201 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:25:07.977179 (+ 978us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:25:07.977180 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:25:07.977206 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:25:07.977233 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:25:07.977412 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:25:07.977414 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:25:07.977416 (+ 2us) client_negotiation.cc:769] Sending connection context
0618 23:25:07.977465 (+ 49us) client_negotiation.cc:240] Negotiation successful
0618 23:25:07.977514 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":86,"threads_started":1}
I20260618 23:25:07.978237 18410 heartbeater.cc:347] Connected to a master server at 127.8.39.190:40183
I20260618 23:25:07.978312 18410 heartbeater.cc:464] Registering TS with master...
I20260618 23:25:07.978453 18410 heartbeater.cc:511] Master 127.8.39.190:40183 requested a full tablet report, sending...
I20260618 23:25:07.978883 18091 ts_manager.cc:194] Registered new tserver with Master: 392c0caef3804f7ba1ff8706fad710b6 (127.8.39.130:37549)
I20260618 23:25:07.980075 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 18282
I20260618 23:25:07.980213 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance
I20260618 23:25:07.980386 8350 external_mini_cluster.cc:1428] Setting key c6656ce3a3c7d0cba8c5d4d4265463d8
I20260618 23:25:07.981302 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.131:0
--local_ip_for_outbound_sockets=127.8.39.131
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:40183
--builtin_ntp_servers=127.8.39.148:46375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260618 23:25:08.060743 18413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:25:08.060942 18413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:25:08.060966 18413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:25:08.062145 18413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:25:08.062186 18413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:25:08.062213 18413 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.131
I20260618 23:25:08.063750 18413 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:46375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:40183
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18413
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.131
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:25:08.064028 18413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:25:08.064296 18413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:25:08.069453 18421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:25:08.069444 18419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:25:08.069445 18418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:25:08.069460 18413 server_base.cc:1034] running on GCE node
I20260618 23:25:08.069845 18413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:25:08.070055 18413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:25:08.071210 18413 hybrid_clock.cc:648] HybridClock initialized: now 1781825108071201 us; error 29 us; skew 500 ppm
I20260618 23:25:08.072494 18413 webserver.cc:460] Webserver started at http://127.8.39.131:34535/ using document root <none> and password file <none>
I20260618 23:25:08.072742 18413 fs_manager.cc:359] Metadata directory not provided
I20260618 23:25:08.072788 18413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:25:08.072947 18413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:25:08.074084 18413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/data/instance:
uuid: "4c31dee9591c4190b2e9a3e636d9e6de"
format_stamp: "Formatted at 2026-06-18 23:25:08 on dist-test-slave-47qt"
server_key: "99361790dfbac9f29941a9e25c432eca"
server_key_iv: "c59e45d91fe2ff0c577ccee96891a202"
server_key_version: "encryptionkey@0"
I20260618 23:25:08.074426 18413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance:
uuid: "4c31dee9591c4190b2e9a3e636d9e6de"
format_stamp: "Formatted at 2026-06-18 23:25:08 on dist-test-slave-47qt"
server_key: "99361790dfbac9f29941a9e25c432eca"
server_key_iv: "c59e45d91fe2ff0c577ccee96891a202"
server_key_version: "encryptionkey@0"
I20260618 23:25:08.075670 18413 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:25:08.076398 18427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:25:08.076563 18413 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260618 23:25:08.076648 18413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/wal
uuid: "4c31dee9591c4190b2e9a3e636d9e6de"
format_stamp: "Formatted at 2026-06-18 23:25:08 on dist-test-slave-47qt"
server_key: "99361790dfbac9f29941a9e25c432eca"
server_key_iv: "c59e45d91fe2ff0c577ccee96891a202"
server_key_version: "encryptionkey@0"
I20260618 23:25:08.076728 18413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:25:08.079911 18413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:25:08.080152 18413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:25:08.080291 18413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:25:08.080505 18413 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:25:08.080868 18413 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:25:08.080929 18413 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:25:08.080977 18413 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:25:08.081003 18413 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:25:08.087260 18413 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.131:44091
I20260618 23:25:08.087390 18540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.131:44091 every 8 connection(s)
I20260618 23:25:08.087692 18413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
I20260618 23:25:08.090698 18141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:08.088437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:34487 (local address 127.8.39.190:40183)
0618 23:25:08.088476 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:08.088478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:08.088698 (+ 220us) server_negotiation.cc:407] Connection header received
0618 23:25:08.089195 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:08.089196 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:08.089208 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:08.089272 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:08.090481 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:25:08.090482 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:25:08.090483 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:25:08.090486 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:25:08.090503 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:25:08.090507 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:25:08.090584 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0618 23:25:08.090635 (+ 51us) server_negotiation.cc:299] Negotiation successful
0618 23:25:08.090648 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260618 23:25:08.090937 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:08.088435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:40183 (local address 127.8.39.131:34487)
0618 23:25:08.088591 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:08.088620 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:08.089054 (+ 434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:08.089322 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:08.089325 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:08.089332 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:25:08.090322 (+ 990us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:25:08.090323 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:25:08.090351 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:25:08.090377 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:25:08.090585 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:25:08.090588 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:25:08.090590 (+ 2us) client_negotiation.cc:769] Sending connection context
0618 23:25:08.090643 (+ 53us) client_negotiation.cc:240] Negotiation successful
0618 23:25:08.090688 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":83,"threads_started":1}
I20260618 23:25:08.091583 18541 heartbeater.cc:347] Connected to a master server at 127.8.39.190:40183
I20260618 23:25:08.091668 18541 heartbeater.cc:464] Registering TS with master...
I20260618 23:25:08.091768 18541 heartbeater.cc:511] Master 127.8.39.190:40183 requested a full tablet report, sending...
I20260618 23:25:08.092139 18091 ts_manager.cc:194] Registered new tserver with Master: 4c31dee9591c4190b2e9a3e636d9e6de (127.8.39.131:44091)
I20260618 23:25:08.095901 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 18413
I20260618 23:25:08.096017 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance
I20260618 23:25:08.096199 8350 external_mini_cluster.cc:1428] Setting key b31c3dbaf590e3d8b36b83c8766904e0
I20260618 23:25:08.097241 8350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260618 23:25:08.098987 18141 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:25:08.098569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36106 (local address 127.8.39.190:40183)
0618 23:25:08.098604 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:08.098605 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:08.098722 (+ 117us) server_negotiation.cc:407] Connection header received
0618 23:25:08.098758 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:08.098759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:08.098763 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:08.098810 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:08.098909 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36106: BlockingRecv error: recv got EOF from 127.0.0.1:36106 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
I20260618 23:25:08.099175 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 18151
I20260618 23:25:08.104136 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 18282
I20260618 23:25:08.108345 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 18413
I20260618 23:25:08.112709 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 18060
2026-06-18T23:25:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (544 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:25:08 dist-test-slave-47qt krb5kdc[18551](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:25:08 dist-test-slave-47qt krb5kdc[18551](info): set up 2 sockets
Jun 18 23:25:08 dist-test-slave-47qt krb5kdc[18551](info): commencing operation
krb5kdc: starting...
W20260618 23:25:09.178088 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:25:09 dist-test-slave-47qt krb5kdc[18551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:25:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:25:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.39.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
I20260618 23:25:09.295313 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:45859
--webserver_interface=127.8.39.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:35883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:45859
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:25:09.373726 18564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:25:09.373967 18564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:25:09.374006 18564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:25:09.375123 18564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:25:09.375190 18564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:25:09.375216 18564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:25:09.375254 18564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:25:09.375300 18564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:25:09.376760 18564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:35883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:45859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:45859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:25:09.377018 18564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:25:09.377249 18564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:25:09.382253 18569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:25:09.382337 18572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:25:09.382402 18564 server_base.cc:1034] running on GCE node
W20260618 23:25:09.382400 18570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:25:09.382720 18564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:25:09.383005 18564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:25:09.384172 18564 hybrid_clock.cc:648] HybridClock initialized: now 1781825109384161 us; error 29 us; skew 500 ppm
Jun 18 23:25:09 dist-test-slave-47qt krb5kdc[18551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825109, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:25:09.386685 18564 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:25:09.387081 18564 webserver.cc:460] Webserver started at http://127.8.39.190:45569/ using document root <none> and password file <none>
I20260618 23:25:09.387270 18564 fs_manager.cc:359] Metadata directory not provided
I20260618 23:25:09.387313 18564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:25:09.387403 18564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:25:09.388432 18564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "3988ede1411048699a4aebbdfbb06348"
format_stamp: "Formatted at 2026-06-18 23:25:09 on dist-test-slave-47qt"
server_key: "49126c0549a67f41970b272f01e69226"
server_key_iv: "17c8bf00b68b18b458d756905c98d5fa"
server_key_version: "encryptionkey@0"
I20260618 23:25:09.388903 18564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "3988ede1411048699a4aebbdfbb06348"
format_stamp: "Formatted at 2026-06-18 23:25:09 on dist-test-slave-47qt"
server_key: "49126c0549a67f41970b272f01e69226"
server_key_iv: "17c8bf00b68b18b458d756905c98d5fa"
server_key_version: "encryptionkey@0"
I20260618 23:25:09.390379 18564 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:25:09.391139 18579 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:25:09.391337 18564 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:25:09.391427 18564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "3988ede1411048699a4aebbdfbb06348"
format_stamp: "Formatted at 2026-06-18 23:25:09 on dist-test-slave-47qt"
server_key: "49126c0549a67f41970b272f01e69226"
server_key_iv: "17c8bf00b68b18b458d756905c98d5fa"
server_key_version: "encryptionkey@0"
I20260618 23:25:09.391484 18564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:25:09.402778 18564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:25:09.404598 18564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:25:09.404748 18564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:25:09.408774 18564 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:45859
I20260618 23:25:09.409070 18631 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:45859 every 8 connection(s)
I20260618 23:25:09.409183 18564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:25:09.409648 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 18564
I20260618 23:25:09.409762 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:25:09.409987 8350 external_mini_cluster.cc:1428] Setting key 6338462f638c556bbd210d052bccb80c
I20260618 23:25:09.410368 18632 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:25:09.413579 18632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3988ede1411048699a4aebbdfbb06348: Bootstrap starting.
I20260618 23:25:09.414147 18632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3988ede1411048699a4aebbdfbb06348: Neither blocks nor log segments found. Creating new log.
I20260618 23:25:09.414471 18632 log.cc:824] T 00000000000000000000000000000000 P 3988ede1411048699a4aebbdfbb06348: Log is configured to *not* fsync() on all Append() calls
I20260618 23:25:09.415243 18632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3988ede1411048699a4aebbdfbb06348: No bootstrap required, opened a new log
Jun 18 23:25:09 dist-test-slave-47qt krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825109, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:25:09.417774 18632 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3988ede1411048699a4aebbdfbb06348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3988ede1411048699a4aebbdfbb06348" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 45859 } }
I20260618 23:25:09.417953 18632 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3988ede1411048699a4aebbdfbb06348 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:25:09.417997 18632 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3988ede1411048699a4aebbdfbb06348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3988ede1411048699a4aebbdfbb06348, State: Initialized, Role: FOLLOWER
I20260618 23:25:09.418126 18632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3988ede1411048699a4aebbdfbb06348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3988ede1411048699a4aebbdfbb06348" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 45859 } }
I20260618 23:25:09.418220 18632 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3988ede1411048699a4aebbdfbb06348 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:25:09.418263 18632 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3988ede1411048699a4aebbdfbb06348 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:25:09.418321 18632 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3988ede1411048699a4aebbdfbb06348 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:25:09.419361 18632 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3988ede1411048699a4aebbdfbb06348 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3988ede1411048699a4aebbdfbb06348" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 45859 } }
I20260618 23:25:09.419605 18632 leader_election.cc:302] T 00000000000000000000000000000000 P 3988ede1411048699a4aebbdfbb06348 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3988ede1411048699a4aebbdfbb06348; no voters:
I20260618 23:25:09.419852 18632 leader_election.cc:288] T 00000000000000000000000000000000 P 3988ede1411048699a4aebbdfbb06348 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:25:09.419936 18637 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3988ede1411048699a4aebbdfbb06348 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:25:09.420105 18637 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3988ede1411048699a4aebbdfbb06348 [term 1 LEADER]: Becoming Leader. State: Replica: 3988ede1411048699a4aebbdfbb06348, State: Running, Role: LEADER
I20260618 23:25:09.420215 18632 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3988ede1411048699a4aebbdfbb06348 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:25:09.420255 18637 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3988ede1411048699a4aebbdfbb06348 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3988ede1411048699a4aebbdfbb06348" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 45859 } }
I20260618 23:25:09.420697 18638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3988ede1411048699a4aebbdfbb06348 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3988ede1411048699a4aebbdfbb06348" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3988ede1411048699a4aebbdfbb06348" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 45859 } } }
I20260618 23:25:09.420799 18638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3988ede1411048699a4aebbdfbb06348 [sys.catalog]: This master's current role is: LEADER
I20260618 23:25:09.421026 18639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3988ede1411048699a4aebbdfbb06348 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3988ede1411048699a4aebbdfbb06348. Latest consensus state: current_term: 1 leader_uuid: "3988ede1411048699a4aebbdfbb06348" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3988ede1411048699a4aebbdfbb06348" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 45859 } } }
I20260618 23:25:09.421317 18639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3988ede1411048699a4aebbdfbb06348 [sys.catalog]: This master's current role is: LEADER
I20260618 23:25:09.421535 18644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:25:09.422042 18644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:25:09.422552 18635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:09.410507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53944 (local address 127.8.39.190:45859)
0618 23:25:09.410743 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:09.410752 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:09.410781 (+ 29us) server_negotiation.cc:407] Connection header received
0618 23:25:09.411463 (+ 682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:09.411471 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:09.411529 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:09.411674 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:09.412278 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:09.413610 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:09.414389 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:09.415011 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:09.415095 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:09.417694 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:25:09.417699 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:25:09.417699 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:25:09.417711 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:25:09.420481 (+ 2770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:25:09.420947 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:25:09.420950 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:25:09.420951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:25:09.421058 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:25:09.421322 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:25:09.421323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:25:09.421323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:25:09.421661 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:25:09.421827 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0618 23:25:09.422311 (+ 484us) server_negotiation.cc:299] Negotiation successful
0618 23:25:09.422417 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":66,"threads_started":1}
I20260618 23:25:09.424253 18644 catalog_manager.cc:1347] Generated new cluster ID: b4b4a7ba2b184e2ca4760b3fecb48959
I20260618 23:25:09.424309 18644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:25:09.438737 18644 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:25:09.439428 18644 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:25:09.455103 18644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3988ede1411048699a4aebbdfbb06348: Generated new TSK 0
I20260618 23:25:09.455471 18644 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.39.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
I20260618 23:25:09.484414 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.129:0
--local_ip_for_outbound_sockets=127.8.39.129
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:45859
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:35883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:25:09.571206 18658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:25:09.571434 18658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:25:09.571462 18658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:25:09.572932 18658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:25:09.572978 18658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:25:09.573009 18658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.129
I20260618 23:25:09.574608 18658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:35883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:45859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.129
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:25:09.574841 18658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:25:09.575071 18658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 23:25:09.580060 18658 server_base.cc:1034] running on GCE node
W20260618 23:25:09.579978 18664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:25:09.580010 18663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:25:09.579981 18666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:25:09.580448 18658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:25:09.580685 18658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:25:09.581863 18658 hybrid_clock.cc:648] HybridClock initialized: now 1781825109581846 us; error 29 us; skew 500 ppm
Jun 18 23:25:09 dist-test-slave-47qt krb5kdc[18551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825109, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:25:09.584692 18658 init.cc:377] Logged in from keytab as kudu/127.8.39.129@KRBTEST.COM (short username kudu)
I20260618 23:25:09.585270 18658 webserver.cc:460] Webserver started at http://127.8.39.129:46795/ using document root <none> and password file <none>
I20260618 23:25:09.585501 18658 fs_manager.cc:359] Metadata directory not provided
I20260618 23:25:09.585565 18658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:25:09.585700 18658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:25:09.586733 18658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/data/instance:
uuid: "78bdb2128df24487905277f7851454b0"
format_stamp: "Formatted at 2026-06-18 23:25:09 on dist-test-slave-47qt"
server_key: "5cf21d2767dd29dda7c33039a869e393"
server_key_iv: "bb62b331257511a875b1f000256f5d9f"
server_key_version: "encryptionkey@0"
I20260618 23:25:09.587119 18658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance:
uuid: "78bdb2128df24487905277f7851454b0"
format_stamp: "Formatted at 2026-06-18 23:25:09 on dist-test-slave-47qt"
server_key: "5cf21d2767dd29dda7c33039a869e393"
server_key_iv: "bb62b331257511a875b1f000256f5d9f"
server_key_version: "encryptionkey@0"
I20260618 23:25:09.588460 18658 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:25:09.589241 18673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:25:09.589432 18658 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:25:09.589531 18658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/wal
uuid: "78bdb2128df24487905277f7851454b0"
format_stamp: "Formatted at 2026-06-18 23:25:09 on dist-test-slave-47qt"
server_key: "5cf21d2767dd29dda7c33039a869e393"
server_key_iv: "bb62b331257511a875b1f000256f5d9f"
server_key_version: "encryptionkey@0"
I20260618 23:25:09.589619 18658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:25:09.620993 18658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:25:09.623189 18658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:25:09.623309 18658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:25:09.623486 18658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:25:09.624001 18658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:25:09.624161 18658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:25:09.624212 18658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:25:09.624250 18658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:25:09.632141 18658 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.129:42003
I20260618 23:25:09.632212 18786 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.129:42003 every 8 connection(s)
I20260618 23:25:09.632560 18658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
Jun 18 23:25:09 dist-test-slave-47qt krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825109, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:25:09.640002 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 18658
I20260618 23:25:09.640105 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance
I20260618 23:25:09.640280 8350 external_mini_cluster.cc:1428] Setting key 76d8370d4df703f78de91a138243c9b9
I20260618 23:25:09.643095 18635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:09.633442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:33059 (local address 127.8.39.190:45859)
0618 23:25:09.633493 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:09.633495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:09.633683 (+ 188us) server_negotiation.cc:407] Connection header received
0618 23:25:09.634216 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:09.634217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:09.634229 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:09.634283 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:09.635206 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:09.635710 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:09.637182 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:09.637417 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:09.637497 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:09.639571 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:25:09.639573 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:25:09.639574 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:25:09.639582 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:25:09.641699 (+ 2117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:25:09.642138 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:25:09.642139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:25:09.642140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:25:09.642198 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:25:09.642447 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:25:09.642448 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:25:09.642448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:25:09.642565 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:25:09.642639 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 23:25:09.642915 (+ 276us) server_negotiation.cc:299] Negotiation successful
0618 23:25:09.643000 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260618 23:25:09.643307 18789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:09.633379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:45859 (local address 127.8.39.129:33059)
0618 23:25:09.633564 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:09.633600 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:09.634115 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:09.634360 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:09.634364 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:09.634922 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:25:09.635110 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:09.635114 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:09.635798 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:09.635800 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:09.637073 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:09.637076 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:09.637509 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:09.637510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:09.637634 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:09.638163 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:25:09.638171 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:25:09.639479 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:25:09.641802 (+ 2323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:25:09.641816 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:25:09.641819 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:25:09.642045 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:25:09.642279 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:25:09.642280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:25:09.642280 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:25:09.642382 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:25:09.642629 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:25:09.642635 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:25:09.642771 (+ 136us) client_negotiation.cc:769] Sending connection context
0618 23:25:09.642896 (+ 125us) client_negotiation.cc:240] Negotiation successful
0618 23:25:09.643059 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":69,"threads_started":1}
I20260618 23:25:09.643895 18787 heartbeater.cc:347] Connected to a master server at 127.8.39.190:45859
I20260618 23:25:09.643994 18787 heartbeater.cc:464] Registering TS with master...
I20260618 23:25:09.644188 18787 heartbeater.cc:511] Master 127.8.39.190:45859 requested a full tablet report, sending...
I20260618 23:25:09.644757 18596 ts_manager.cc:194] Registered new tserver with Master: 78bdb2128df24487905277f7851454b0 (127.8.39.129:42003)
I20260618 23:25:09.645936 18596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.129@KRBTEST.COM'} at 127.8.39.129:33059
WARNING: no policy specified for kudu/127.8.39.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
I20260618 23:25:09.667491 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.130:0
--local_ip_for_outbound_sockets=127.8.39.130
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:45859
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:35883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:25:09.747977 18792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:25:09.748172 18792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:25:09.748194 18792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:25:09.749325 18792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:25:09.749364 18792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:25:09.749389 18792 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.130
I20260618 23:25:09.750722 18792 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:35883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:45859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.130
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:25:09.750905 18792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:25:09.751066 18792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:25:09.755919 18800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:25:09.755930 18797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:25:09.755957 18792 server_base.cc:1034] running on GCE node
W20260618 23:25:09.755962 18798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:25:09.756305 18792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:25:09.756541 18792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:25:09.757779 18792 hybrid_clock.cc:648] HybridClock initialized: now 1781825109757765 us; error 42 us; skew 500 ppm
Jun 18 23:25:09 dist-test-slave-47qt krb5kdc[18551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825109, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:25:09.761303 18792 init.cc:377] Logged in from keytab as kudu/127.8.39.130@KRBTEST.COM (short username kudu)
I20260618 23:25:09.761787 18792 webserver.cc:460] Webserver started at http://127.8.39.130:37753/ using document root <none> and password file <none>
I20260618 23:25:09.762005 18792 fs_manager.cc:359] Metadata directory not provided
I20260618 23:25:09.762069 18792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:25:09.762224 18792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:25:09.763249 18792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/data/instance:
uuid: "540d9bf4e77d4f628925de6e8c3d683a"
format_stamp: "Formatted at 2026-06-18 23:25:09 on dist-test-slave-47qt"
server_key: "dc96ad939d8da0be6322cf06dd8661fc"
server_key_iv: "6e5422aab3a14bd42df590e21ef9f886"
server_key_version: "encryptionkey@0"
I20260618 23:25:09.763605 18792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance:
uuid: "540d9bf4e77d4f628925de6e8c3d683a"
format_stamp: "Formatted at 2026-06-18 23:25:09 on dist-test-slave-47qt"
server_key: "dc96ad939d8da0be6322cf06dd8661fc"
server_key_iv: "6e5422aab3a14bd42df590e21ef9f886"
server_key_version: "encryptionkey@0"
I20260618 23:25:09.765033 18792 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:25:09.766009 18807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:25:09.766350 18792 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:25:09.766479 18792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/wal
uuid: "540d9bf4e77d4f628925de6e8c3d683a"
format_stamp: "Formatted at 2026-06-18 23:25:09 on dist-test-slave-47qt"
server_key: "dc96ad939d8da0be6322cf06dd8661fc"
server_key_iv: "6e5422aab3a14bd42df590e21ef9f886"
server_key_version: "encryptionkey@0"
I20260618 23:25:09.766561 18792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:25:09.784192 18792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:25:09.786991 18792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:25:09.787245 18792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:25:09.787474 18792 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:25:09.787773 18792 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:25:09.787824 18792 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:25:09.787873 18792 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:25:09.787899 18792 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:25:09.794168 18792 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.130:45225
I20260618 23:25:09.794210 18920 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.130:45225 every 8 connection(s)
I20260618 23:25:09.794595 18792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
Jun 18 23:25:09 dist-test-slave-47qt krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825109, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:25:09.802703 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 18792
I20260618 23:25:09.802831 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance
I20260618 23:25:09.803006 8350 external_mini_cluster.cc:1428] Setting key f6bc87b9b7a78a944908e52cf7ac4bd6
I20260618 23:25:09.805231 18635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:09.795343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:52103 (local address 127.8.39.190:45859)
0618 23:25:09.795420 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:09.795423 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:09.795640 (+ 217us) server_negotiation.cc:407] Connection header received
0618 23:25:09.796199 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:09.796200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:09.796213 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:09.796274 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:09.797521 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:09.798014 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:09.798989 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:09.799271 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:09.799385 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:09.801638 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:25:09.801640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:25:09.801640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:25:09.801648 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:25:09.803730 (+ 2082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:25:09.804194 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:25:09.804195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:25:09.804195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:25:09.804264 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:25:09.804530 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:25:09.804531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:25:09.804531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:25:09.804662 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:25:09.804752 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0618 23:25:09.805022 (+ 270us) server_negotiation.cc:299] Negotiation successful
0618 23:25:09.805106 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260618 23:25:09.805444 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:09.795302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:45859 (local address 127.8.39.130:52103)
0618 23:25:09.795501 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:09.795538 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:09.796074 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:09.796344 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:09.796348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:09.797210 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:25:09.797418 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:09.797423 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:09.798105 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:09.798106 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:09.798900 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:09.798902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:09.799396 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:09.799397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:09.799526 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:09.800014 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:25:09.800022 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:25:09.801526 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:25:09.803830 (+ 2304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:25:09.803838 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:25:09.803840 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:25:09.804089 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:25:09.804348 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:25:09.804349 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:25:09.804350 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:25:09.804457 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:25:09.804745 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:25:09.804751 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:25:09.804889 (+ 138us) client_negotiation.cc:769] Sending connection context
0618 23:25:09.805001 (+ 112us) client_negotiation.cc:240] Negotiation successful
0618 23:25:09.805202 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":77,"threads_started":1}
I20260618 23:25:09.805912 18921 heartbeater.cc:347] Connected to a master server at 127.8.39.190:45859
I20260618 23:25:09.806007 18921 heartbeater.cc:464] Registering TS with master...
I20260618 23:25:09.806203 18921 heartbeater.cc:511] Master 127.8.39.190:45859 requested a full tablet report, sending...
I20260618 23:25:09.806595 18596 ts_manager.cc:194] Registered new tserver with Master: 540d9bf4e77d4f628925de6e8c3d683a (127.8.39.130:45225)
I20260618 23:25:09.807121 18596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.130@KRBTEST.COM'} at 127.8.39.130:52103
WARNING: no policy specified for kudu/127.8.39.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
I20260618 23:25:09.828297 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.131:0
--local_ip_for_outbound_sockets=127.8.39.131
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:45859
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:35883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:25:09.909034 18926 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:25:09.909333 18926 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:25:09.909371 18926 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:25:09.910529 18926 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:25:09.910595 18926 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:25:09.910641 18926 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.131
I20260618 23:25:09.912089 18926 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:35883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:45859
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18926
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.131
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:25:09.912351 18926 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:25:09.912569 18926 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:25:09.917896 18932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:25:09.918121 18926 server_base.cc:1034] running on GCE node
W20260618 23:25:09.917879 18934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:25:09.917884 18931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:25:09.918465 18926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:25:09.918709 18926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:25:09.919902 18926 hybrid_clock.cc:648] HybridClock initialized: now 1781825109919842 us; error 79 us; skew 500 ppm
Jun 18 23:25:09 dist-test-slave-47qt krb5kdc[18551](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825109, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:25:09.923369 18926 init.cc:377] Logged in from keytab as kudu/127.8.39.131@KRBTEST.COM (short username kudu)
I20260618 23:25:09.923957 18926 webserver.cc:460] Webserver started at http://127.8.39.131:41401/ using document root <none> and password file <none>
I20260618 23:25:09.924233 18926 fs_manager.cc:359] Metadata directory not provided
I20260618 23:25:09.924283 18926 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:25:09.924371 18926 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:25:09.925468 18926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/data/instance:
uuid: "bc084555088b43208d367afb1e2c5a0b"
format_stamp: "Formatted at 2026-06-18 23:25:09 on dist-test-slave-47qt"
server_key: "c1ea101d8d8bce17e5cc25d3b363d461"
server_key_iv: "173f056acc71542d628ac61cab5b2d27"
server_key_version: "encryptionkey@0"
I20260618 23:25:09.926040 18926 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance:
uuid: "bc084555088b43208d367afb1e2c5a0b"
format_stamp: "Formatted at 2026-06-18 23:25:09 on dist-test-slave-47qt"
server_key: "c1ea101d8d8bce17e5cc25d3b363d461"
server_key_iv: "173f056acc71542d628ac61cab5b2d27"
server_key_version: "encryptionkey@0"
I20260618 23:25:09.928027 18926 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260618 23:25:09.928791 18941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:25:09.929006 18926 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:25:09.929309 18926 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/wal
uuid: "bc084555088b43208d367afb1e2c5a0b"
format_stamp: "Formatted at 2026-06-18 23:25:09 on dist-test-slave-47qt"
server_key: "c1ea101d8d8bce17e5cc25d3b363d461"
server_key_iv: "173f056acc71542d628ac61cab5b2d27"
server_key_version: "encryptionkey@0"
I20260618 23:25:09.929383 18926 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:25:09.943885 18926 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:25:09.945796 18926 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:25:09.945909 18926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:25:09.946081 18926 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:25:09.946332 18926 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:25:09.946364 18926 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:25:09.946385 18926 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:25:09.946398 18926 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:25:09.952306 18926 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.131:46675
I20260618 23:25:09.952343 19054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.131:46675 every 8 connection(s)
I20260618 23:25:09.952706 18926 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
I20260618 23:25:09.953449 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 18926
I20260618 23:25:09.953540 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance
I20260618 23:25:09.953739 8350 external_mini_cluster.cc:1428] Setting key ebc03a37a7a1e43dcfe60ff99949fe4b
Jun 18 23:25:09 dist-test-slave-47qt krb5kdc[18551](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825109, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:25:09.962532 18635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:09.953638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:49813 (local address 127.8.39.190:45859)
0618 23:25:09.953695 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:09.953701 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:09.953933 (+ 232us) server_negotiation.cc:407] Connection header received
0618 23:25:09.954565 (+ 632us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:09.954566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:09.954577 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:09.954632 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:09.955664 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:09.956184 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:09.957205 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:09.957394 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:09.957477 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:09.959526 (+ 2049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:25:09.959527 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:25:09.959527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:25:09.959535 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:25:09.961021 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:25:09.961545 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:25:09.961546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:25:09.961546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:25:09.961617 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:25:09.961883 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:25:09.961884 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:25:09.961885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:25:09.962001 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:25:09.962083 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0618 23:25:09.962348 (+ 265us) server_negotiation.cc:299] Negotiation successful
0618 23:25:09.962431 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260618 23:25:09.962738 19057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:09.953613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:45859 (local address 127.8.39.131:49813)
0618 23:25:09.953800 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:09.953848 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:09.954462 (+ 614us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:09.954702 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:09.954706 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:09.955383 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:25:09.955568 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:09.955573 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:09.956291 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:09.956292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:09.957075 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:09.957077 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:09.957496 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:09.957497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:09.957614 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:09.958116 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:25:09.958124 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:25:09.959434 (+ 1310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:25:09.961204 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:25:09.961211 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:25:09.961213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:25:09.961448 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:25:09.961703 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:25:09.961704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:25:09.961704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:25:09.961811 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:25:09.962081 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:25:09.962088 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:25:09.962217 (+ 129us) client_negotiation.cc:769] Sending connection context
0618 23:25:09.962328 (+ 111us) client_negotiation.cc:240] Negotiation successful
0618 23:25:09.962487 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":87,"threads_started":1}
I20260618 23:25:09.963215 19055 heartbeater.cc:347] Connected to a master server at 127.8.39.190:45859
I20260618 23:25:09.963300 19055 heartbeater.cc:464] Registering TS with master...
I20260618 23:25:09.963438 19055 heartbeater.cc:511] Master 127.8.39.190:45859 requested a full tablet report, sending...
I20260618 23:25:09.963783 18596 ts_manager.cc:194] Registered new tserver with Master: bc084555088b43208d367afb1e2c5a0b (127.8.39.131:46675)
I20260618 23:25:09.964188 18596 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.131@KRBTEST.COM'} at 127.8.39.131:49813
I20260618 23:25:09.965606 8350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:25:09.972769 18635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:09.966930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53950 (local address 127.8.39.190:45859)
0618 23:25:09.967008 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:09.967010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:09.967016 (+ 6us) server_negotiation.cc:407] Connection header received
0618 23:25:09.967032 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:09.967033 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:09.967040 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:09.967128 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:25:09.967697 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:09.968164 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:09.968883 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:09.969068 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:09.969166 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:09.970320 (+ 1154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:25:09.970321 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:25:09.970321 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:25:09.970338 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:25:09.971606 (+ 1268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:25:09.971981 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:25:09.971982 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:25:09.971983 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:25:09.972037 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:25:09.972229 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:25:09.972230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:25:09.972230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:25:09.972369 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:25:09.972469 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0618 23:25:09.972578 (+ 109us) server_negotiation.cc:299] Negotiation successful
0618 23:25:09.972652 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 23:25:09.974128 18596 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53950:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:25:09.975581 18596 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:25:09.982311 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:09.978224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:45225 (local address 127.0.0.1:42246)
0618 23:25:09.978381 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:09.978395 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:09.978447 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:09.978940 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:09.978941 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:09.978943 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:25:09.978988 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:09.978989 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:09.979895 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:09.979896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:09.981028 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:09.981030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:09.982015 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:09.982016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:09.982098 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:09.982111 (+ 13us) client_negotiation.cc:769] Sending connection context
0618 23:25:09.982182 (+ 71us) client_negotiation.cc:240] Negotiation successful
0618 23:25:09.982234 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":94,"threads_started":1}
I20260618 23:25:09.982317 19071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:09.978607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42246 (local address 127.8.39.130:45225)
0618 23:25:09.978751 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:09.978755 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:09.978765 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:25:09.978807 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:09.978809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:09.978824 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:09.978939 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:25:09.979048 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:09.979786 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:09.981132 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:09.981893 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:09.982012 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:09.982073 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0618 23:25:09.982196 (+ 123us) server_negotiation.cc:299] Negotiation successful
0618 23:25:09.982242 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":81,"threads_started":1}
I20260618 23:25:09.982478 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:09.977987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:46675 (local address 127.0.0.1:44724)
0618 23:25:09.978108 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:09.978144 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:09.978228 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:09.978653 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:09.978656 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:09.978659 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:25:09.978728 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:09.978729 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:09.980360 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:09.980361 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:09.981372 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:09.981374 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:09.982271 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:09.982272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:09.982333 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:09.982338 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:25:09.982371 (+ 33us) client_negotiation.cc:240] Negotiation successful
0618 23:25:09.982410 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":31,"thread_start_us":64,"threads_started":1}
I20260618 23:25:09.982774 19068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:09.978289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44724 (local address 127.8.39.131:46675)
0618 23:25:09.978457 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:09.978462 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:09.978473 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:25:09.978517 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:09.978519 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:09.978535 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:09.978841 (+ 306us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:25:09.979165 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:09.980248 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:09.981453 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:09.982161 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:09.982547 (+ 386us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:09.982608 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0618 23:25:09.982634 (+ 26us) server_negotiation.cc:299] Negotiation successful
0618 23:25:09.982684 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
I20260618 23:25:09.982893 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:09.978445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:42003 (local address 127.0.0.1:58968)
0618 23:25:09.978587 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:09.978596 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:09.978637 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:09.979063 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:09.979063 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:09.979066 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:25:09.979116 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:09.979118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:09.980376 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:09.980377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:09.981440 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:09.981442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:09.982353 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:09.982354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:09.982722 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:09.982726 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:25:09.982772 (+ 46us) client_negotiation.cc:240] Negotiation successful
0618 23:25:09.982815 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
I20260618 23:25:09.983170 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:09.978567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58968 (local address 127.8.39.129:42003)
0618 23:25:09.978854 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:09.978858 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:09.978867 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:25:09.978904 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:09.978906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:09.978920 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:09.978994 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:25:09.979181 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:09.980254 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:09.981510 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:09.982195 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:09.982260 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:09.982322 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0618 23:25:09.983024 (+ 702us) server_negotiation.cc:299] Negotiation successful
0618 23:25:09.983069 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":253,"thread_start_us":67,"threads_started":1}
I20260618 23:25:09.983280 18855 tablet_service.cc:1467] Processing CreateTablet for tablet cee3b9f01a9d4693930097c6a88738ab (DEFAULT_TABLE table=test-table [id=72b6eb5a77b44bbab17eb6f054b472e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:25:09.983510 18989 tablet_service.cc:1467] Processing CreateTablet for tablet cee3b9f01a9d4693930097c6a88738ab (DEFAULT_TABLE table=test-table [id=72b6eb5a77b44bbab17eb6f054b472e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:25:09.983618 18855 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cee3b9f01a9d4693930097c6a88738ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:25:09.983860 18989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cee3b9f01a9d4693930097c6a88738ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:25:09.985836 18721 tablet_service.cc:1467] Processing CreateTablet for tablet cee3b9f01a9d4693930097c6a88738ab (DEFAULT_TABLE table=test-table [id=72b6eb5a77b44bbab17eb6f054b472e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:25:09.986189 18721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cee3b9f01a9d4693930097c6a88738ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:25:09.986426 19072 tablet_bootstrap.cc:492] T cee3b9f01a9d4693930097c6a88738ab P 540d9bf4e77d4f628925de6e8c3d683a: Bootstrap starting.
I20260618 23:25:09.986956 19073 tablet_bootstrap.cc:492] T cee3b9f01a9d4693930097c6a88738ab P bc084555088b43208d367afb1e2c5a0b: Bootstrap starting.
I20260618 23:25:09.987131 19072 tablet_bootstrap.cc:654] T cee3b9f01a9d4693930097c6a88738ab P 540d9bf4e77d4f628925de6e8c3d683a: Neither blocks nor log segments found. Creating new log.
I20260618 23:25:09.987656 19073 tablet_bootstrap.cc:654] T cee3b9f01a9d4693930097c6a88738ab P bc084555088b43208d367afb1e2c5a0b: Neither blocks nor log segments found. Creating new log.
I20260618 23:25:09.987702 19072 log.cc:824] T cee3b9f01a9d4693930097c6a88738ab P 540d9bf4e77d4f628925de6e8c3d683a: Log is configured to *not* fsync() on all Append() calls
I20260618 23:25:09.988157 19073 log.cc:824] T cee3b9f01a9d4693930097c6a88738ab P bc084555088b43208d367afb1e2c5a0b: Log is configured to *not* fsync() on all Append() calls
I20260618 23:25:09.988502 19072 tablet_bootstrap.cc:492] T cee3b9f01a9d4693930097c6a88738ab P 540d9bf4e77d4f628925de6e8c3d683a: No bootstrap required, opened a new log
I20260618 23:25:09.988585 19072 ts_tablet_manager.cc:1397] T cee3b9f01a9d4693930097c6a88738ab P 540d9bf4e77d4f628925de6e8c3d683a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260618 23:25:09.988909 19073 tablet_bootstrap.cc:492] T cee3b9f01a9d4693930097c6a88738ab P bc084555088b43208d367afb1e2c5a0b: No bootstrap required, opened a new log
I20260618 23:25:09.989019 19073 ts_tablet_manager.cc:1397] T cee3b9f01a9d4693930097c6a88738ab P bc084555088b43208d367afb1e2c5a0b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260618 23:25:09.989689 19076 tablet_bootstrap.cc:492] T cee3b9f01a9d4693930097c6a88738ab P 78bdb2128df24487905277f7851454b0: Bootstrap starting.
I20260618 23:25:09.990422 19076 tablet_bootstrap.cc:654] T cee3b9f01a9d4693930097c6a88738ab P 78bdb2128df24487905277f7851454b0: Neither blocks nor log segments found. Creating new log.
I20260618 23:25:09.990625 19073 raft_consensus.cc:348] T cee3b9f01a9d4693930097c6a88738ab P bc084555088b43208d367afb1e2c5a0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc084555088b43208d367afb1e2c5a0b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 46675 } } peers { permanent_uuid: "540d9bf4e77d4f628925de6e8c3d683a" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 45225 } } peers { permanent_uuid: "78bdb2128df24487905277f7851454b0" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 42003 } }
I20260618 23:25:09.990787 19073 raft_consensus.cc:374] T cee3b9f01a9d4693930097c6a88738ab P bc084555088b43208d367afb1e2c5a0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:25:09.990855 19073 raft_consensus.cc:729] T cee3b9f01a9d4693930097c6a88738ab P bc084555088b43208d367afb1e2c5a0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc084555088b43208d367afb1e2c5a0b, State: Initialized, Role: FOLLOWER
I20260618 23:25:09.990993 19076 log.cc:824] T cee3b9f01a9d4693930097c6a88738ab P 78bdb2128df24487905277f7851454b0: Log is configured to *not* fsync() on all Append() calls
I20260618 23:25:09.990978 19073 consensus_queue.cc:260] T cee3b9f01a9d4693930097c6a88738ab P bc084555088b43208d367afb1e2c5a0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc084555088b43208d367afb1e2c5a0b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 46675 } } peers { permanent_uuid: "540d9bf4e77d4f628925de6e8c3d683a" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 45225 } } peers { permanent_uuid: "78bdb2128df24487905277f7851454b0" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 42003 } }
I20260618 23:25:09.990927 19072 raft_consensus.cc:348] T cee3b9f01a9d4693930097c6a88738ab P 540d9bf4e77d4f628925de6e8c3d683a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc084555088b43208d367afb1e2c5a0b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 46675 } } peers { permanent_uuid: "540d9bf4e77d4f628925de6e8c3d683a" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 45225 } } peers { permanent_uuid: "78bdb2128df24487905277f7851454b0" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 42003 } }
I20260618 23:25:09.991074 19072 raft_consensus.cc:374] T cee3b9f01a9d4693930097c6a88738ab P 540d9bf4e77d4f628925de6e8c3d683a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:25:09.991142 19072 raft_consensus.cc:729] T cee3b9f01a9d4693930097c6a88738ab P 540d9bf4e77d4f628925de6e8c3d683a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 540d9bf4e77d4f628925de6e8c3d683a, State: Initialized, Role: FOLLOWER
I20260618 23:25:09.991258 19073 ts_tablet_manager.cc:1428] T cee3b9f01a9d4693930097c6a88738ab P bc084555088b43208d367afb1e2c5a0b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260618 23:25:09.991298 19055 heartbeater.cc:503] Master 127.8.39.190:45859 was elected leader, sending a full tablet report...
I20260618 23:25:09.991356 19072 consensus_queue.cc:260] T cee3b9f01a9d4693930097c6a88738ab P 540d9bf4e77d4f628925de6e8c3d683a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc084555088b43208d367afb1e2c5a0b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 46675 } } peers { permanent_uuid: "540d9bf4e77d4f628925de6e8c3d683a" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 45225 } } peers { permanent_uuid: "78bdb2128df24487905277f7851454b0" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 42003 } }
I20260618 23:25:09.991626 18921 heartbeater.cc:503] Master 127.8.39.190:45859 was elected leader, sending a full tablet report...
I20260618 23:25:09.991657 19072 ts_tablet_manager.cc:1428] T cee3b9f01a9d4693930097c6a88738ab P 540d9bf4e77d4f628925de6e8c3d683a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:25:09.992326 19076 tablet_bootstrap.cc:492] T cee3b9f01a9d4693930097c6a88738ab P 78bdb2128df24487905277f7851454b0: No bootstrap required, opened a new log
I20260618 23:25:09.992420 19076 ts_tablet_manager.cc:1397] T cee3b9f01a9d4693930097c6a88738ab P 78bdb2128df24487905277f7851454b0: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 23:25:09.994496 19076 raft_consensus.cc:348] T cee3b9f01a9d4693930097c6a88738ab P 78bdb2128df24487905277f7851454b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc084555088b43208d367afb1e2c5a0b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 46675 } } peers { permanent_uuid: "540d9bf4e77d4f628925de6e8c3d683a" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 45225 } } peers { permanent_uuid: "78bdb2128df24487905277f7851454b0" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 42003 } }
I20260618 23:25:09.994781 19076 raft_consensus.cc:374] T cee3b9f01a9d4693930097c6a88738ab P 78bdb2128df24487905277f7851454b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:25:09.994823 19076 raft_consensus.cc:729] T cee3b9f01a9d4693930097c6a88738ab P 78bdb2128df24487905277f7851454b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78bdb2128df24487905277f7851454b0, State: Initialized, Role: FOLLOWER
I20260618 23:25:09.995008 19076 consensus_queue.cc:260] T cee3b9f01a9d4693930097c6a88738ab P 78bdb2128df24487905277f7851454b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc084555088b43208d367afb1e2c5a0b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 46675 } } peers { permanent_uuid: "540d9bf4e77d4f628925de6e8c3d683a" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 45225 } } peers { permanent_uuid: "78bdb2128df24487905277f7851454b0" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 42003 } }
I20260618 23:25:09.995409 19076 ts_tablet_manager.cc:1428] T cee3b9f01a9d4693930097c6a88738ab P 78bdb2128df24487905277f7851454b0: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260618 23:25:09.995445 18787 heartbeater.cc:503] Master 127.8.39.190:45859 was elected leader, sending a full tablet report...
W20260618 23:25:10.045195 18922 tablet.cc:2369] T cee3b9f01a9d4693930097c6a88738ab P 540d9bf4e77d4f628925de6e8c3d683a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 23:25:10.133285 18788 tablet.cc:2369] T cee3b9f01a9d4693930097c6a88738ab P 78bdb2128df24487905277f7851454b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:25:10.142536 19080 raft_consensus.cc:482] T cee3b9f01a9d4693930097c6a88738ab P 78bdb2128df24487905277f7851454b0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:25:10.142639 19080 raft_consensus.cc:504] T cee3b9f01a9d4693930097c6a88738ab P 78bdb2128df24487905277f7851454b0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc084555088b43208d367afb1e2c5a0b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 46675 } } peers { permanent_uuid: "540d9bf4e77d4f628925de6e8c3d683a" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 45225 } } peers { permanent_uuid: "78bdb2128df24487905277f7851454b0" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 42003 } }
I20260618 23:25:10.143062 19080 leader_election.cc:288] T cee3b9f01a9d4693930097c6a88738ab P 78bdb2128df24487905277f7851454b0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc084555088b43208d367afb1e2c5a0b (127.8.39.131:46675), 540d9bf4e77d4f628925de6e8c3d683a (127.8.39.130:45225)
I20260618 23:25:10.146126 18789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:10.143269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:46675 (local address 127.8.39.129:57021)
0618 23:25:10.143331 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:10.143345 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:10.143395 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:10.143564 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:10.143566 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:10.143567 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:25:10.143645 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:10.143647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:10.144382 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:10.144383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:10.145247 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:10.145250 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:10.145853 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:10.145854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:10.145932 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:10.145936 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:25:10.146000 (+ 64us) client_negotiation.cc:240] Negotiation successful
0618 23:25:10.146041 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260618 23:25:10.146186 19068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:10.143367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:57021 (local address 127.8.39.131:46675)
0618 23:25:10.143429 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:10.143431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:10.143441 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:25:10.143471 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:10.143472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:10.143482 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:10.143541 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:25:10.143717 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:10.144245 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:10.145318 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:10.145778 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:10.145835 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:10.145842 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0618 23:25:10.146042 (+ 200us) server_negotiation.cc:299] Negotiation successful
0618 23:25:10.146116 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260618 23:25:10.146425 19071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:10.143306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:34285 (local address 127.8.39.130:45225)
0618 23:25:10.143368 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:10.143370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:10.143488 (+ 118us) server_negotiation.cc:407] Connection header received
0618 23:25:10.143579 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:10.143580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:10.143593 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:10.143650 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:25:10.143813 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:10.144407 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:10.145567 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:10.146035 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:10.146100 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:10.146109 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0618 23:25:10.146308 (+ 199us) server_negotiation.cc:299] Negotiation successful
0618 23:25:10.146356 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260618 23:25:10.146494 19009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee3b9f01a9d4693930097c6a88738ab" candidate_uuid: "78bdb2128df24487905277f7851454b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc084555088b43208d367afb1e2c5a0b" is_pre_election: true
I20260618 23:25:10.146660 19081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:10.143269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:45225 (local address 127.8.39.129:34285)
0618 23:25:10.143415 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:10.143425 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:25:10.143490 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:25:10.143664 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:25:10.143665 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:25:10.143666 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:25:10.143747 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:10.143748 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:10.144541 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:10.144542 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:10.145480 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:25:10.145482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:25:10.146112 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:25:10.146113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:25:10.146191 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:10.146195 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:25:10.146513 (+ 318us) client_negotiation.cc:240] Negotiation successful
0618 23:25:10.146570 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":22,"thread_start_us":58,"threads_started":1}
I20260618 23:25:10.146675 19009 raft_consensus.cc:2454] T cee3b9f01a9d4693930097c6a88738ab P bc084555088b43208d367afb1e2c5a0b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78bdb2128df24487905277f7851454b0 in term 0.
I20260618 23:25:10.146935 18674 leader_election.cc:302] T cee3b9f01a9d4693930097c6a88738ab P 78bdb2128df24487905277f7851454b0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78bdb2128df24487905277f7851454b0, bc084555088b43208d367afb1e2c5a0b; no voters:
I20260618 23:25:10.147051 19080 raft_consensus.cc:2781] T cee3b9f01a9d4693930097c6a88738ab P 78bdb2128df24487905277f7851454b0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:25:10.147007 18875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee3b9f01a9d4693930097c6a88738ab" candidate_uuid: "78bdb2128df24487905277f7851454b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "540d9bf4e77d4f628925de6e8c3d683a" is_pre_election: true
I20260618 23:25:10.147104 19080 raft_consensus.cc:482] T cee3b9f01a9d4693930097c6a88738ab P 78bdb2128df24487905277f7851454b0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:25:10.147125 19080 raft_consensus.cc:3037] T cee3b9f01a9d4693930097c6a88738ab P 78bdb2128df24487905277f7851454b0 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:25:10.147154 18875 raft_consensus.cc:2454] T cee3b9f01a9d4693930097c6a88738ab P 540d9bf4e77d4f628925de6e8c3d683a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 78bdb2128df24487905277f7851454b0 in term 0.
I20260618 23:25:10.148038 19080 raft_consensus.cc:504] T cee3b9f01a9d4693930097c6a88738ab P 78bdb2128df24487905277f7851454b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc084555088b43208d367afb1e2c5a0b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 46675 } } peers { permanent_uuid: "540d9bf4e77d4f628925de6e8c3d683a" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 45225 } } peers { permanent_uuid: "78bdb2128df24487905277f7851454b0" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 42003 } }
I20260618 23:25:10.148191 19080 leader_election.cc:288] T cee3b9f01a9d4693930097c6a88738ab P 78bdb2128df24487905277f7851454b0 [CANDIDATE]: Term 1 election: Requested vote from peers bc084555088b43208d367afb1e2c5a0b (127.8.39.131:46675), 540d9bf4e77d4f628925de6e8c3d683a (127.8.39.130:45225)
I20260618 23:25:10.148339 19009 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee3b9f01a9d4693930097c6a88738ab" candidate_uuid: "78bdb2128df24487905277f7851454b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc084555088b43208d367afb1e2c5a0b"
I20260618 23:25:10.148415 19009 raft_consensus.cc:3037] T cee3b9f01a9d4693930097c6a88738ab P bc084555088b43208d367afb1e2c5a0b [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:25:10.149333 19009 raft_consensus.cc:2454] T cee3b9f01a9d4693930097c6a88738ab P bc084555088b43208d367afb1e2c5a0b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78bdb2128df24487905277f7851454b0 in term 1.
I20260618 23:25:10.149451 18875 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cee3b9f01a9d4693930097c6a88738ab" candidate_uuid: "78bdb2128df24487905277f7851454b0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "540d9bf4e77d4f628925de6e8c3d683a"
I20260618 23:25:10.149501 18674 leader_election.cc:302] T cee3b9f01a9d4693930097c6a88738ab P 78bdb2128df24487905277f7851454b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 78bdb2128df24487905277f7851454b0, bc084555088b43208d367afb1e2c5a0b; no voters:
I20260618 23:25:10.149528 18875 raft_consensus.cc:3037] T cee3b9f01a9d4693930097c6a88738ab P 540d9bf4e77d4f628925de6e8c3d683a [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:25:10.149575 19080 raft_consensus.cc:2781] T cee3b9f01a9d4693930097c6a88738ab P 78bdb2128df24487905277f7851454b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:25:10.149725 19080 raft_consensus.cc:686] T cee3b9f01a9d4693930097c6a88738ab P 78bdb2128df24487905277f7851454b0 [term 1 LEADER]: Becoming Leader. State: Replica: 78bdb2128df24487905277f7851454b0, State: Running, Role: LEADER
I20260618 23:25:10.149842 19080 consensus_queue.cc:237] T cee3b9f01a9d4693930097c6a88738ab P 78bdb2128df24487905277f7851454b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc084555088b43208d367afb1e2c5a0b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 46675 } } peers { permanent_uuid: "540d9bf4e77d4f628925de6e8c3d683a" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 45225 } } peers { permanent_uuid: "78bdb2128df24487905277f7851454b0" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 42003 } }
I20260618 23:25:10.150357 18875 raft_consensus.cc:2454] T cee3b9f01a9d4693930097c6a88738ab P 540d9bf4e77d4f628925de6e8c3d683a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 78bdb2128df24487905277f7851454b0 in term 1.
I20260618 23:25:10.150632 18595 catalog_manager.cc:5495] T cee3b9f01a9d4693930097c6a88738ab P 78bdb2128df24487905277f7851454b0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 78bdb2128df24487905277f7851454b0 (127.8.39.129). New cstate: current_term: 1 leader_uuid: "78bdb2128df24487905277f7851454b0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc084555088b43208d367afb1e2c5a0b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 46675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "540d9bf4e77d4f628925de6e8c3d683a" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 45225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "78bdb2128df24487905277f7851454b0" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 42003 } health_report { overall_health: HEALTHY } } }
I20260618 23:25:10.166889 19070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:10.164564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58974 (local address 127.8.39.129:42003)
0618 23:25:10.164639 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0618 23:25:10.164641 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:25:10.164649 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:25:10.164684 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:25:10.164685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:25:10.164701 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:25:10.164759 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:25:10.164918 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:10.165416 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:10.166168 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:25:10.166346 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:25:10.166401 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:25:10.166518 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:25:10.166624 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:25:10.166744 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0618 23:25:10.166760 (+ 16us) server_negotiation.cc:299] Negotiation successful
0618 23:25:10.166776 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260618 23:25:10.168526 19009 raft_consensus.cc:1261] T cee3b9f01a9d4693930097c6a88738ab P bc084555088b43208d367afb1e2c5a0b [term 1 FOLLOWER]: Refusing update from remote peer 78bdb2128df24487905277f7851454b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:25:10.168541 18875 raft_consensus.cc:1261] T cee3b9f01a9d4693930097c6a88738ab P 540d9bf4e77d4f628925de6e8c3d683a [term 1 FOLLOWER]: Refusing update from remote peer 78bdb2128df24487905277f7851454b0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:25:10.168953 19082 consensus_queue.cc:1035] T cee3b9f01a9d4693930097c6a88738ab P 78bdb2128df24487905277f7851454b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc084555088b43208d367afb1e2c5a0b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 46675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:25:10.169077 19080 consensus_queue.cc:1035] T cee3b9f01a9d4693930097c6a88738ab P 78bdb2128df24487905277f7851454b0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "540d9bf4e77d4f628925de6e8c3d683a" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 45225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:25:10.173472 18596 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:25:10.173579 18596 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53950:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:25:10.174243 18596 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3988ede1411048699a4aebbdfbb06348: Sending DeleteTablet for 3 replicas of tablet cee3b9f01a9d4693930097c6a88738ab
I20260618 23:25:10.174597 18989 tablet_service.cc:1514] Processing DeleteTablet for tablet cee3b9f01a9d4693930097c6a88738ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:25:10 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:44724
I20260618 23:25:10.174612 18855 tablet_service.cc:1514] Processing DeleteTablet for tablet cee3b9f01a9d4693930097c6a88738ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:25:10 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:42246
I20260618 23:25:10.174607 18721 tablet_service.cc:1514] Processing DeleteTablet for tablet cee3b9f01a9d4693930097c6a88738ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:25:10 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:58968
I20260618 23:25:10.174870 19096 tablet_replica.cc:330] T cee3b9f01a9d4693930097c6a88738ab P bc084555088b43208d367afb1e2c5a0b: stopping tablet replica
I20260618 23:25:10.174970 19096 raft_consensus.cc:2229] T cee3b9f01a9d4693930097c6a88738ab P bc084555088b43208d367afb1e2c5a0b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:25:10.175033 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 18658
I20260618 23:25:10.175069 19096 raft_consensus.cc:2258] T cee3b9f01a9d4693930097c6a88738ab P bc084555088b43208d367afb1e2c5a0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:25:10.175220 19097 tablet_replica.cc:330] T cee3b9f01a9d4693930097c6a88738ab P 540d9bf4e77d4f628925de6e8c3d683a: stopping tablet replica
I20260618 23:25:10.175302 19097 raft_consensus.cc:2229] T cee3b9f01a9d4693930097c6a88738ab P 540d9bf4e77d4f628925de6e8c3d683a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:25:10.175385 19097 raft_consensus.cc:2258] T cee3b9f01a9d4693930097c6a88738ab P 540d9bf4e77d4f628925de6e8c3d683a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:25:10.175861 19096 ts_tablet_manager.cc:1905] T cee3b9f01a9d4693930097c6a88738ab P bc084555088b43208d367afb1e2c5a0b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:25:10.177315 19097 ts_tablet_manager.cc:1905] T cee3b9f01a9d4693930097c6a88738ab P 540d9bf4e77d4f628925de6e8c3d683a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:25:10.177794 19096 ts_tablet_manager.cc:1918] T cee3b9f01a9d4693930097c6a88738ab P bc084555088b43208d367afb1e2c5a0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:25:10.177870 19096 log.cc:1196] T cee3b9f01a9d4693930097c6a88738ab P bc084555088b43208d367afb1e2c5a0b: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-2/wal/wals/cee3b9f01a9d4693930097c6a88738ab
I20260618 23:25:10.178144 19096 ts_tablet_manager.cc:1939] T cee3b9f01a9d4693930097c6a88738ab P bc084555088b43208d367afb1e2c5a0b: Deleting consensus metadata
I20260618 23:25:10.178624 18580 catalog_manager.cc:4841] TS bc084555088b43208d367afb1e2c5a0b (127.8.39.131:46675): tablet cee3b9f01a9d4693930097c6a88738ab (table test-table [id=72b6eb5a77b44bbab17eb6f054b472e3]) successfully deleted
I20260618 23:25:10.179685 19097 ts_tablet_manager.cc:1918] T cee3b9f01a9d4693930097c6a88738ab P 540d9bf4e77d4f628925de6e8c3d683a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:25:10.179939 19097 log.cc:1196] T cee3b9f01a9d4693930097c6a88738ab P 540d9bf4e77d4f628925de6e8c3d683a: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781825053542692-8350-0/minicluster-data/ts-1/wal/wals/cee3b9f01a9d4693930097c6a88738ab
I20260618 23:25:10.180308 19097 ts_tablet_manager.cc:1939] T cee3b9f01a9d4693930097c6a88738ab P 540d9bf4e77d4f628925de6e8c3d683a: Deleting consensus metadata
I20260618 23:25:10.180739 18583 catalog_manager.cc:4841] TS 540d9bf4e77d4f628925de6e8c3d683a (127.8.39.130:45225): tablet cee3b9f01a9d4693930097c6a88738ab (table test-table [id=72b6eb5a77b44bbab17eb6f054b472e3]) successfully deleted
W20260618 23:25:10.182205 18581 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.39.129:42003 (error 108)
I20260618 23:25:10.182572 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 18792
I20260618 23:25:10.182754 19069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:25:10.182606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:42003 (local address 127.0.0.1:58988)
0618 23:25:10.182687 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0618 23:25:10.182721 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.129:42003: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260618 23:25:10.183109 18581 catalog_manager.cc:4584] TS 78bdb2128df24487905277f7851454b0 (127.8.39.129:42003): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cee3b9f01a9d4693930097c6a88738ab: Network error: Client connection negotiation failed: client connection to 127.8.39.129:42003: connect: Connection refused (error 111)
I20260618 23:25:10.189020 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 18926
I20260618 23:25:10.193542 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 18564
2026-06-18T23:25:10Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2081 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:25:10 dist-test-slave-47qt krb5kdc[19101](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:25:10 dist-test-slave-47qt krb5kdc[19101](info): set up 2 sockets
Jun 18 23:25:10 dist-test-slave-47qt krb5kdc[19101](info): commencing operation
krb5kdc: starting...
W20260618 23:25:11.250588 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:25:11 dist-test-slave-47qt krb5kdc[19101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825111, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:25:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:25:11Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.39.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.39.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.39.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825053542692-8350-0/krb5kdc/rangeradmin_127.8.39.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.39.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825053542692-8350-0/krb5kdc/rangeradmin_127.8.39.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.39.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.39.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.39.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825053542692-8350-0/krb5kdc/rangerlookup_127.8.39.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.39.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825053542692-8350-0/krb5kdc/rangerlookup_127.8.39.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.39.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.39.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.39.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825053542692-8350-0/krb5kdc/HTTP_127.8.39.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.39.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825053542692-8350-0/krb5kdc/HTTP_127.8.39.148@KRBTEST.COM.keytab.
I20260618 23:25:11.412067 8350 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825053542692-8350-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskNmhGNv/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825053542692-8350-0/minicluster-data/postgres -l logfile start
2026-06-18 23:25:14.004 UTC [19131] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-18 23:25:14.004 UTC [19131] LOG: listening on IPv4 address "127.8.39.148", port 43683
2026-06-18 23:25:14.007 UTC [19131] LOG: listening on Unix socket "/tmp/.s.PGSQL.43683"
2026-06-18 23:25:14.042 UTC [19134] LOG: database system was shut down at 2026-06-18 23:25:12 UTC
2026-06-18 23:25:14.046 UTC [19131] LOG: database system is ready to accept connections
I20260618 23:25:15.018067 8350 mini_postgres.cc:96] Postgres bound to 43683
2026-06-18 23:25:15.023 UTC [19142] FATAL: database "slave" does not exist
127.8.39.148:43683 - accepting connections
I20260618 23:25:15.024231 8350 mini_ranger.cc:161] Starting Ranger...
I20260618 23:25:15.039233 8350 mini_ranger.cc:84] Created miniranger Postgres user
I20260618 23:25:15.540642 8350 mini_ranger.cc:87] Created ranger Postgres database
I20260618 23:25:15.540705 8350 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825053542692-8350-0/minicluster-data/ranger-admin
2026-06-18 23:25:15,813 [I] DB FLAVOR :POSTGRES
2026-06-18 23:25:15,814 [I] --------- Verifying Ranger DB connection ---------
2026-06-18 23:25:15,814 [I] Checking connection..
2026-06-18 23:25:15,814 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:43683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-18 23:25:16,076 [I] Checking connection passed.
2026-06-18 23:25:16,076 [I] --------- Verifying version history table ---------
2026-06-18 23:25:16,076 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:43683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-18 23:25:16,319 [I] Table x_db_version_h does not exist in database ranger
2026-06-18 23:25:16,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:43683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-18 23:25:16,553 [I] Table x_db_version_h does not exist in database ranger
2026-06-18 23:25:16,553 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-18 23:25:16,553 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:43683/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-18 23:25:16.791 UTC [19206] WARNING: there is no transaction in progress
2026-06-18 23:25:16,796 [I] create_dbversion_catalog.sql file imported successfully
2026-06-18 23:25:16,797 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:43683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-18 23:25:17,044 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-18 23:25:17,044 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-18 23:25:17,044 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:43683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-18 23:25:17,299 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:43683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-18 23:25:17,539 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:43683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-47qt.c.gcp-upstream.internal','N') ;"
2026-06-18 23:25:17,786 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:43683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-18 23:25:18,040 [I] Table x_portal_user does not exist in database ranger
2026-06-18 23:25:18,040 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:43683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-18 23:25:18,276 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-18 23:25:18,276 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:43683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-18 23:25:18,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:43683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-18 23:25:18,772 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-18 23:25:18,772 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:43683/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-18 23:25:19.750 UTC [19314] WARNING: there is no transaction in progress
2026-06-18 23:25:19.767 UTC [19314] WARNING: there is no transaction in progress
2026-06-18 23:25:19.785 UTC [19314] WARNING: there is no transaction in progress
2026-06-18 23:25:19.802 UTC [19314] WARNING: there is no transaction in progress
2026-06-18 23:25:19.820 UTC [19314] WARNING: there is no transaction in progress
2026-06-18 23:25:19.841 UTC [19314] WARNING: there is no transaction in progress
2026-06-18 23:25:19.908 UTC [19314] WARNING: there is no transaction in progress
2026-06-18 23:25:19.916 UTC [19314] WARNING: there is no transaction in progress
2026-06-18 23:25:19.930 UTC [19314] WARNING: there is no transaction in progress
2026-06-18 23:25:19.940 UTC [19314] WARNING: there is no transaction in progress
2026-06-18 23:25:19.950 UTC [19314] WARNING: there is no transaction in progress
2026-06-18 23:25:19.965 UTC [19314] WARNING: there is no transaction in progress
2026-06-18 23:25:19.975 UTC [19314] WARNING: there is no transaction in progress
2026-06-18 23:25:19.982 UTC [19314] WARNING: there is no transaction in progress
2026-06-18 23:25:19.989 UTC [19314] WARNING: there is no transaction in progress
2026-06-18 23:25:19.996 UTC [19314] WARNING: there is no transaction in progress
2026-06-18 23:25:20,647 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-18 23:25:20,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:43683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-18 23:25:20,877 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-18 23:25:20,877 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:43683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-18 23:25:21,136 [I] Table x_portal_user already exists in database 'ranger'
2026-06-18 23:25:21,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:43683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-18 23:25:21,384 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-18 23:25:21,384 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:43683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-18 23:25:21,632 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:43683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-18 23:25:21,893 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:43683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-47qt.c.gcp-upstream.internal';"
2026-06-18 23:25:22,147 [I] CORE_DB_SCHEMA import status has been updated
2026-06-18 23:25:22,147 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:43683/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-18 23:25:22,398 [I] DB_PATCHES have already been applied
I20260618 23:25:22.403450 8350 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825053542692-8350-0/minicluster-data/ranger-admin:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskNmhGNv/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260618 23:25:22.403508 8350 mini_ranger.cc:193] Using host: 127.8.39.148
I20260618 23:25:22.406507 8350 mini_ranger.cc:227] Ranger admin URL: http://127.8.39.148:58145
Jun 18, 2026 11:25:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 18, 2026 11:25:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 18, 2026 11:25:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825053542692-8350-0/minicluster-data/ranger-admin/ews/webapp
Jun 18, 2026 11:25:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825053542692-8350-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 18, 2026 11:25:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825053542692-8350-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 18, 2026 11:25:23 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825053542692-8350-0/minicluster-data/ranger-admin/ews/webapp].
Jun 18, 2026 11:25:23 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260618 23:25:23.513458 8350 mini_ranger.cc:160] Time spent starting Ranger: real 8.489s user 0.000s sys 0.008s
Jun 18, 2026 11:25:43 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 18, 2026 11:25:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 18, 2026 11:25:43 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 18, 2026 11:25:43 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 18, 2026 11:25:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 18, 2026 11:25:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 18, 2026 11:25:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 18, 2026 11:25:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 18, 2026 11:25:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 18, 2026 11:25:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 18, 2026 11:25:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 18, 2026 11:25:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 18, 2026 11:25:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 18, 2026 11:25:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 18, 2026 11:25:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 18, 2026 11:25:43 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 18, 2026 11:25:43 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 18, 2026 11:25:44 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260618 23:25:44.561087 8350 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.39.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.39.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.39.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825053542692-8350-0/krb5kdc/rangerkms_127.8.39.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.39.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825053542692-8350-0/krb5kdc/rangerkms_127.8.39.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.39.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.39.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.39.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825053542692-8350-0/krb5kdc/HTTP_127.8.39.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.39.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825053542692-8350-0/krb5kdc/HTTP_127.8.39.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 18 23:25:44 dist-test-slave-47qt krb5kdc[19101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825144, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260618 23:25:44.642249 8350 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260618 23:25:44.652088 8350 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260618 23:25:45.680923 8350 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260618 23:25:45.680990 8350 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825053542692-8350-0/minicluster-data/ranger-kms
I20260618 23:25:45.681003 8350 mini_ranger_kms.cc:227] Using postgres at 127.8.39.148:43683
2026-06-18 23:25:45,865 [I] DB FLAVOR :POSTGRES
2026-06-18 23:25:45,866 [I] --------- Verifying Ranger DB connection ---------
2026-06-18 23:25:45,866 [I] Checking connection
2026-06-18 23:25:45,866 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825053542692-8350-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:43683/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-18 23:25:46,146 [I] connection success
2026-06-18 23:25:46,146 [I] --------- Verifying Ranger DB tables ---------
2026-06-18 23:25:46,146 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-18 23:25:46,147 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825053542692-8350-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:43683/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-18 23:25:46,393 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-18 23:25:46,393 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-18 23:25:46,393 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-18 23:25:46,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825053542692-8350-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:43683/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825053542692-8350-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-18 23:25:46,665 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260618 23:25:47.122638 8350 mini_ranger_kms.cc:310] Created kms service
I20260618 23:25:47.355509 8350 mini_ranger_kms.cc:326] Created kudu user
I20260618 23:25:47.401101 8350 mini_ranger_kms.cc:343] Created rangerkms user
Jun 18 23:25:47 dist-test-slave-47qt krb5kdc[19101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825144, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.39.148@KRBTEST.COM
I20260618 23:25:47.821525 8350 mini_ranger_kms.cc:384] Added ranger policy
I20260618 23:25:47.821641 8350 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825053542692-8350-0/minicluster-data/ranger-kms:/tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260618 23:25:47.821664 8350 mini_ranger_kms.cc:240] Using host: 127.8.39.148
I20260618 23:25:47.825227 8350 mini_ranger_kms.cc:276] Ranger KMS PID: 19494
I20260618 23:25:47.825374 8350 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.39.148:37657
Jun 18, 2026 11:25:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 18, 2026 11:25:48 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 18, 2026 11:25:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825053542692-8350-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 18, 2026 11:25:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825053542692-8350-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 18, 2026 11:25:49 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825053542692-8350-0/minicluster-data/ranger-kms/ews/webapp].
Jun 18, 2026 11:25:49 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Jun 18 23:25:50 dist-test-slave-47qt krb5kdc[19101](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781825150, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.39.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 18 23:25:50 dist-test-slave-47qt krb5kdc[19101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825150, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.39.148@KRBTEST.COM for HTTP/127.8.39.148@KRBTEST.COM
Jun 18 23:25:51 dist-test-slave-47qt krb5kdc[19101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825150, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.39.148@KRBTEST.COM for HTTP/127.8.39.148@KRBTEST.COM
Jun 18 23:26:21 dist-test-slave-47qt krb5kdc[19101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825150, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.39.148@KRBTEST.COM for HTTP/127.8.39.148@KRBTEST.COM
Jun 18 23:26:21 dist-test-slave-47qt krb5kdc[19101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825150, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.39.148@KRBTEST.COM for HTTP/127.8.39.148@KRBTEST.COM
Jun 18 23:26:51 dist-test-slave-47qt krb5kdc[19101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825150, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.39.148@KRBTEST.COM for HTTP/127.8.39.148@KRBTEST.COM
Jun 18 23:26:51 dist-test-slave-47qt krb5kdc[19101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825150, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.39.148@KRBTEST.COM for HTTP/127.8.39.148@KRBTEST.COM
Jun 18 23:27:21 dist-test-slave-47qt krb5kdc[19101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825150, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.39.148@KRBTEST.COM for HTTP/127.8.39.148@KRBTEST.COM
Jun 18 23:27:21 dist-test-slave-47qt krb5kdc[19101](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825150, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.39.148@KRBTEST.COM for HTTP/127.8.39.148@KRBTEST.COM
I20260618 23:27:48.763322 8350 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 124.121s user 0.030s sys 0.272s
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:1013: Failure
Failed
Bad status: Timed out: Failed to start the Ranger KMS service: process with PID 19494 is not yet bound to any port at the specified addresses; last attempt running lsof returned 'Not found: could not find pattern of a bound port in lsof output: p19494
f196
n127.0.0.1:36244->127.8.39.148:43683'
I20260618 23:27:48.764211 8350 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260618 23:27:48.784897 8350 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260618 23:27:48.784992 8350 mini_ranger.cc:66] Stopping Ranger...
I20260618 23:27:49.129951 8350 mini_ranger.cc:68] Stopped Ranger
2026-06-18 23:27:49.130 UTC [19131] LOG: received smart shutdown request
2026-06-18 23:27:49.133 UTC [19131] LOG: background worker "logical replication launcher" (PID 19140) exited with exit code 1
2026-06-18 23:27:49.133 UTC [19135] LOG: shutting down
2026-06-18 23:27:49.276 UTC [19131] LOG: database system is shut down
2026-06-18T23:27:49Z chronyd exiting
I20260618 23:27:49.293834 8350 test_util.cc:175] -----------------------------------------------
I20260618 23:27:49.294021 8350 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781825053542692-8350-0
[ FAILED ] SecurityITest.TestEncryptionWithKMSIntegration (159081 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:27:49 dist-test-slave-47qt krb5kdc[19628](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:27:49 dist-test-slave-47qt krb5kdc[19628](info): set up 2 sockets
Jun 18 23:27:49 dist-test-slave-47qt krb5kdc[19628](info): commencing operation
krb5kdc: starting...
W20260618 23:27:50.348495 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:27:50 dist-test-slave-47qt krb5kdc[19628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825270, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:27:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:27:50Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.39.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.39.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.39.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/rangeradmin_127.8.39.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.39.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/rangeradmin_127.8.39.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.39.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.39.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.39.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/rangerlookup_127.8.39.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.39.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/rangerlookup_127.8.39.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.39.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.39.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.39.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/HTTP_127.8.39.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.39.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/HTTP_127.8.39.148@KRBTEST.COM.keytab.
I20260618 23:27:50.520310 8350 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskNmhGNv/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/postgres -l logfile start
2026-06-18 23:27:53.130 UTC [19658] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-18 23:27:53.130 UTC [19658] LOG: listening on IPv4 address "127.8.39.148", port 46199
2026-06-18 23:27:53.133 UTC [19658] LOG: listening on Unix socket "/tmp/.s.PGSQL.46199"
2026-06-18 23:27:53.168 UTC [19661] LOG: database system was shut down at 2026-06-18 23:27:51 UTC
2026-06-18 23:27:53.172 UTC [19658] LOG: database system is ready to accept connections
I20260618 23:27:54.144471 8350 mini_postgres.cc:96] Postgres bound to 46199
2026-06-18 23:27:54.150 UTC [19669] FATAL: database "slave" does not exist
127.8.39.148:46199 - accepting connections
I20260618 23:27:54.151355 8350 mini_ranger.cc:161] Starting Ranger...
I20260618 23:27:54.165798 8350 mini_ranger.cc:84] Created miniranger Postgres user
I20260618 23:27:54.673175 8350 mini_ranger.cc:87] Created ranger Postgres database
I20260618 23:27:54.673261 8350 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ranger-admin
2026-06-18 23:27:54,975 [I] DB FLAVOR :POSTGRES
2026-06-18 23:27:54,976 [I] --------- Verifying Ranger DB connection ---------
2026-06-18 23:27:54,976 [I] Checking connection..
2026-06-18 23:27:54,976 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:46199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-18 23:27:55,230 [I] Checking connection passed.
2026-06-18 23:27:55,231 [I] --------- Verifying version history table ---------
2026-06-18 23:27:55,231 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:46199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-18 23:27:55,469 [I] Table x_db_version_h does not exist in database ranger
2026-06-18 23:27:55,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:46199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-18 23:27:55,721 [I] Table x_db_version_h does not exist in database ranger
2026-06-18 23:27:55,722 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-18 23:27:55,722 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:46199/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-18 23:27:55.966 UTC [19733] WARNING: there is no transaction in progress
2026-06-18 23:27:55,973 [I] create_dbversion_catalog.sql file imported successfully
2026-06-18 23:27:55,973 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:46199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-18 23:27:56,253 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-18 23:27:56,253 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-18 23:27:56,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:46199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-18 23:27:56,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:46199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-18 23:27:56,795 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:46199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-47qt.c.gcp-upstream.internal','N') ;"
2026-06-18 23:27:57,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:46199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-18 23:27:57,326 [I] Table x_portal_user does not exist in database ranger
2026-06-18 23:27:57,327 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:46199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-18 23:27:57,585 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-18 23:27:57,585 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:46199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-18 23:27:57,829 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:46199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-18 23:27:58,083 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-18 23:27:58,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:46199/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-18 23:27:59.072 UTC [19841] WARNING: there is no transaction in progress
2026-06-18 23:27:59.091 UTC [19841] WARNING: there is no transaction in progress
2026-06-18 23:27:59.110 UTC [19841] WARNING: there is no transaction in progress
2026-06-18 23:27:59.127 UTC [19841] WARNING: there is no transaction in progress
2026-06-18 23:27:59.144 UTC [19841] WARNING: there is no transaction in progress
2026-06-18 23:27:59.166 UTC [19841] WARNING: there is no transaction in progress
2026-06-18 23:27:59.233 UTC [19841] WARNING: there is no transaction in progress
2026-06-18 23:27:59.240 UTC [19841] WARNING: there is no transaction in progress
2026-06-18 23:27:59.254 UTC [19841] WARNING: there is no transaction in progress
2026-06-18 23:27:59.267 UTC [19841] WARNING: there is no transaction in progress
2026-06-18 23:27:59.277 UTC [19841] WARNING: there is no transaction in progress
2026-06-18 23:27:59.292 UTC [19841] WARNING: there is no transaction in progress
2026-06-18 23:27:59.302 UTC [19841] WARNING: there is no transaction in progress
2026-06-18 23:27:59.309 UTC [19841] WARNING: there is no transaction in progress
2026-06-18 23:27:59.317 UTC [19841] WARNING: there is no transaction in progress
2026-06-18 23:27:59.324 UTC [19841] WARNING: there is no transaction in progress
2026-06-18 23:28:00,056 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-18 23:28:00,056 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:46199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-18 23:28:00,302 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-18 23:28:00,302 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:46199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-18 23:28:00,566 [I] Table x_portal_user already exists in database 'ranger'
2026-06-18 23:28:00,566 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:46199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-18 23:28:00,831 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-18 23:28:00,832 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:46199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-18 23:28:01,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:46199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-18 23:28:01,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:46199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-47qt.c.gcp-upstream.internal';"
2026-06-18 23:28:01,567 [I] CORE_DB_SCHEMA import status has been updated
2026-06-18 23:28:01,567 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:46199/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-18 23:28:01,826 [I] DB_PATCHES have already been applied
I20260618 23:28:01.833554 8350 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ranger-admin:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskNmhGNv/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260618 23:28:01.833621 8350 mini_ranger.cc:193] Using host: 127.8.39.148
I20260618 23:28:01.836589 8350 mini_ranger.cc:227] Ranger admin URL: http://127.8.39.148:40421
Jun 18, 2026 11:28:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 18, 2026 11:28:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 18, 2026 11:28:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ranger-admin/ews/webapp
Jun 18, 2026 11:28:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 18, 2026 11:28:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 18, 2026 11:28:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ranger-admin/ews/webapp].
Jun 18, 2026 11:28:02 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260618 23:28:02.990082 8350 mini_ranger.cc:160] Time spent starting Ranger: real 8.839s user 0.002s sys 0.007s
Jun 18, 2026 11:28:23 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 18, 2026 11:28:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Jun 18, 2026 11:28:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 18, 2026 11:28:23 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 18, 2026 11:28:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 18, 2026 11:28:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 18, 2026 11:28:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 18, 2026 11:28:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 18, 2026 11:28:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 18, 2026 11:28:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 18, 2026 11:28:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 18, 2026 11:28:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 18, 2026 11:28:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 18, 2026 11:28:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 18, 2026 11:28:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 18, 2026 11:28:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 18, 2026 11:28:23 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 18, 2026 11:28:24 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260618 23:28:24.759498 8350 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.39.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.39.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.39.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/rangerkms_127.8.39.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.39.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/rangerkms_127.8.39.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.39.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.39.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.39.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/HTTP_127.8.39.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.39.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/HTTP_127.8.39.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 18 23:28:24 dist-test-slave-47qt krb5kdc[19628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825304, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260618 23:28:24.847065 8350 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260618 23:28:24.856289 8350 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260618 23:28:25.838781 8350 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260618 23:28:25.838873 8350 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ranger-kms
I20260618 23:28:25.838896 8350 mini_ranger_kms.cc:227] Using postgres at 127.8.39.148:46199
2026-06-18 23:28:26,014 [I] DB FLAVOR :POSTGRES
2026-06-18 23:28:26,014 [I] --------- Verifying Ranger DB connection ---------
2026-06-18 23:28:26,014 [I] Checking connection
2026-06-18 23:28:26,014 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:46199/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-18 23:28:26,275 [I] connection success
2026-06-18 23:28:26,275 [I] --------- Verifying Ranger DB tables ---------
2026-06-18 23:28:26,275 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-18 23:28:26,275 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:46199/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-18 23:28:26,521 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-18 23:28:26,522 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-18 23:28:26,522 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-18 23:28:26,522 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.39.148:46199/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-18 23:28:26,798 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260618 23:28:27.288635 8350 mini_ranger_kms.cc:310] Created kms service
I20260618 23:28:27.488554 8350 mini_ranger_kms.cc:326] Created kudu user
I20260618 23:28:27.534868 8350 mini_ranger_kms.cc:343] Created rangerkms user
Jun 18 23:28:27 dist-test-slave-47qt krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825304, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.39.148@KRBTEST.COM
I20260618 23:28:27.987474 8350 mini_ranger_kms.cc:384] Added ranger policy
I20260618 23:28:27.987591 8350 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ranger-kms:/tmp/dist-test-taskNmhGNv/build/release/bin/postgresql.jar:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskNmhGNv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260618 23:28:27.987610 8350 mini_ranger_kms.cc:240] Using host: 127.8.39.148
I20260618 23:28:27.991421 8350 mini_ranger_kms.cc:276] Ranger KMS PID: 20021
I20260618 23:28:27.991528 8350 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.39.148:42659
Jun 18, 2026 11:28:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 18, 2026 11:28:29 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 18, 2026 11:28:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 18, 2026 11:28:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 18, 2026 11:28:29 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ranger-kms/ews/webapp].
Jun 18, 2026 11:28:29 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260618 23:28:30.311580 8350 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.465s user 0.000s sys 0.156s
I20260618 23:28:30.311694 8350 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260618 23:28:30.311795 8350 mini_ranger_kms.cc:401] 127.8.39.148:42659/kms/v1/keys
Jun 18 23:28:30 dist-test-slave-47qt krb5kdc[19628](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781825310, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.39.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 18 23:28:30 dist-test-slave-47qt krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825310, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.39.148@KRBTEST.COM for HTTP/127.8.39.148@KRBTEST.COM
Jun 18 23:28:31 dist-test-slave-47qt krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825310, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.39.148@KRBTEST.COM for HTTP/127.8.39.148@KRBTEST.COM
Jun 18 23:28:33 dist-test-slave-47qt krb5kdc[19628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.39.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
I20260618 23:28:33.662001 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:44023
--webserver_interface=127.8.39.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:44023
--ranger_config_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.39.148:42659/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:28:33.752327 20072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:28:33.752555 20072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:28:33.752579 20072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:28:33.754570 20072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:28:33.754635 20072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:28:33.754652 20072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:28:33.754664 20072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:28:33.754676 20072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:28:33.756544 20072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.39.148:42659/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:44023
--ranger_config_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:44023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:28:33.756856 20072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:28:33.757144 20072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:28:33.762985 20078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:33.763051 20080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:33.763044 20077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:28:33.763090 20072 server_base.cc:1034] running on GCE node
I20260618 23:28:33.763454 20072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:28:33.763708 20072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:28:33.764933 20072 hybrid_clock.cc:648] HybridClock initialized: now 1781825313764907 us; error 37 us; skew 500 ppm
Jun 18 23:28:33 dist-test-slave-47qt krb5kdc[19628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825313, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:28:33.767448 20072 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:28:33.767906 20072 webserver.cc:460] Webserver started at http://127.8.39.190:38583/ using document root <none> and password file <none>
I20260618 23:28:33.768134 20072 fs_manager.cc:359] Metadata directory not provided
I20260618 23:28:33.768204 20072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:28:33.768357 20072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 23:28:33 dist-test-slave-47qt krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825313, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for HTTP/127.8.39.148@KRBTEST.COM
I20260618 23:28:33.817727 20072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "1cb91418f59d4ce59dcd70082f5996aa"
format_stamp: "Formatted at 2026-06-18 23:28:33 on dist-test-slave-47qt"
server_key: "fa5da2d2d513ec60a4e769fd2cc152c6"
server_key_iv: "cffd3e7935d30a93941d571e4a2478da"
server_key_version: "kuduclusterkey@0"
I20260618 23:28:33.818271 20072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "1cb91418f59d4ce59dcd70082f5996aa"
format_stamp: "Formatted at 2026-06-18 23:28:33 on dist-test-slave-47qt"
server_key: "fa5da2d2d513ec60a4e769fd2cc152c6"
server_key_iv: "cffd3e7935d30a93941d571e4a2478da"
server_key_version: "kuduclusterkey@0"
I20260618 23:28:33.820449 20072 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260618 23:28:33.839174 20087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:33.839555 20072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260618 23:28:33.839710 20072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "1cb91418f59d4ce59dcd70082f5996aa"
format_stamp: "Formatted at 2026-06-18 23:28:33 on dist-test-slave-47qt"
server_key: "fa5da2d2d513ec60a4e769fd2cc152c6"
server_key_iv: "cffd3e7935d30a93941d571e4a2478da"
server_key_version: "kuduclusterkey@0"
I20260618 23:28:33.839811 20072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:28:33.854055 20072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:28:33.857170 20072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:28:33.857336 20072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:28:33.863046 20072 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:44023
I20260618 23:28:33.863073 20139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:44023 every 8 connection(s)
I20260618 23:28:33.863514 20072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:28:33.864239 20140 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:28:33.866602 20140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa: Bootstrap starting.
I20260618 23:28:33.867203 20140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa: Neither blocks nor log segments found. Creating new log.
I20260618 23:28:33.867506 20140 log.cc:824] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa: Log is configured to *not* fsync() on all Append() calls
I20260618 23:28:33.868556 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 20072
I20260618 23:28:33.868654 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:28:33.868781 20140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa: No bootstrap required, opened a new log
I20260618 23:28:33.868857 8350 external_mini_cluster.cc:1428] Setting key d07788f8ff39c64a8ecd43d706eb78ec
I20260618 23:28:33.871853 20140 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb91418f59d4ce59dcd70082f5996aa" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44023 } }
I20260618 23:28:33.872057 20140 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:28:33.872092 20140 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb91418f59d4ce59dcd70082f5996aa, State: Initialized, Role: FOLLOWER
I20260618 23:28:33.872243 20140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb91418f59d4ce59dcd70082f5996aa" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44023 } }
I20260618 23:28:33.872319 20140 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:28:33.872349 20140 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:28:33.872418 20140 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:28:33.873109 20140 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb91418f59d4ce59dcd70082f5996aa" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44023 } }
I20260618 23:28:33.873301 20140 leader_election.cc:302] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cb91418f59d4ce59dcd70082f5996aa; no voters:
I20260618 23:28:33.873493 20140 leader_election.cc:288] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:28:33.873519 20145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:28:33.873729 20145 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [term 1 LEADER]: Becoming Leader. State: Replica: 1cb91418f59d4ce59dcd70082f5996aa, State: Running, Role: LEADER
I20260618 23:28:33.873802 20140 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:28:33.874789 20145 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb91418f59d4ce59dcd70082f5996aa" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44023 } }
I20260618 23:28:33.876189 20140 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260618 23:28:33.876483 20146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cb91418f59d4ce59dcd70082f5996aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb91418f59d4ce59dcd70082f5996aa" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44023 } } }
I20260618 23:28:33.876574 20146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [sys.catalog]: This master's current role is: LEADER
I20260618 23:28:33.876767 20147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cb91418f59d4ce59dcd70082f5996aa. Latest consensus state: current_term: 1 leader_uuid: "1cb91418f59d4ce59dcd70082f5996aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb91418f59d4ce59dcd70082f5996aa" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44023 } } }
I20260618 23:28:33.876791 20160 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 18 23:28:33 dist-test-slave-47qt krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825313, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:28:33.876861 20147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [sys.catalog]: This master's current role is: LEADER
I20260618 23:28:33.877403 20160 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:28:33.884181 20144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:33.869436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58766 (local address 127.8.39.190:44023)
0618 23:28:33.869614 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:33.869623 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:33.869654 (+ 31us) server_negotiation.cc:407] Connection header received
0618 23:28:33.870251 (+ 597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:33.870261 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:33.870334 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:33.870502 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:33.871282 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:33.872677 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:33.873797 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:33.874397 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:33.874486 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:33.877477 (+ 2991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:33.877482 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:33.877482 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:33.877496 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:33.882279 (+ 4783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:33.882744 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:33.882746 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:33.882747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:33.882842 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:33.883057 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:33.883058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:33.883058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:33.883355 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:33.883499 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:33.883616 (+ 117us) server_negotiation.cc:299] Negotiation successful
0618 23:28:33.883830 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":71,"threads_started":1}
I20260618 23:28:33.887884 20160 catalog_manager.cc:1347] Generated new cluster ID: 16dd47ebf5e44aa9b6d90b72a75d75bb
I20260618 23:28:33.887943 20160 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:28:33.923980 20160 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:28:33.924948 20160 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:28:33.937760 20160 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa: Generated new TSK 0
I20260618 23:28:33.938329 20160 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.39.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
I20260618 23:28:34.653420 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.39.148:42659/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.129:0
--local_ip_for_outbound_sockets=127.8.39.129
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:44023
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:28:34.762457 20185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:28:34.762696 20185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:28:34.762737 20185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:28:34.763908 20185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:28:34.764096 20185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:28:34.764178 20185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.129
I20260618 23:28:34.765915 20185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.39.148:42659/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:44023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.129
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:28:34.766203 20185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:28:34.766440 20185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:28:34.772286 20191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:34.772392 20190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:34.772285 20193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:28:34.773406 20185 server_base.cc:1034] running on GCE node
I20260618 23:28:34.773648 20185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:28:34.773945 20185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:28:34.775092 20185 hybrid_clock.cc:648] HybridClock initialized: now 1781825314775069 us; error 35 us; skew 500 ppm
Jun 18 23:28:34 dist-test-slave-47qt krb5kdc[19628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825314, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:28:34.777954 20185 init.cc:377] Logged in from keytab as kudu/127.8.39.129@KRBTEST.COM (short username kudu)
I20260618 23:28:34.778466 20185 webserver.cc:460] Webserver started at http://127.8.39.129:34307/ using document root <none> and password file <none>
I20260618 23:28:34.778726 20185 fs_manager.cc:359] Metadata directory not provided
I20260618 23:28:34.778797 20185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:28:34.778967 20185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 23:28:34 dist-test-slave-47qt krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825314, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for HTTP/127.8.39.148@KRBTEST.COM
I20260618 23:28:34.794435 20185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-0/data/instance:
uuid: "914fcc9ca1d849f882da228c2e701c4b"
format_stamp: "Formatted at 2026-06-18 23:28:34 on dist-test-slave-47qt"
server_key: "9a44590ce3749a9b6e7b29c1110dd002"
server_key_iv: "af754fabd9db7b779e7d7336d1ec7bfa"
server_key_version: "kuduclusterkey@0"
I20260618 23:28:34.794901 20185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance:
uuid: "914fcc9ca1d849f882da228c2e701c4b"
format_stamp: "Formatted at 2026-06-18 23:28:34 on dist-test-slave-47qt"
server_key: "9a44590ce3749a9b6e7b29c1110dd002"
server_key_iv: "af754fabd9db7b779e7d7336d1ec7bfa"
server_key_version: "kuduclusterkey@0"
I20260618 23:28:34.801554 20185 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.001s
I20260618 23:28:34.818243 20202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:34.818559 20185 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:28:34.818672 20185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-0/wal
uuid: "914fcc9ca1d849f882da228c2e701c4b"
format_stamp: "Formatted at 2026-06-18 23:28:34 on dist-test-slave-47qt"
server_key: "9a44590ce3749a9b6e7b29c1110dd002"
server_key_iv: "af754fabd9db7b779e7d7336d1ec7bfa"
server_key_version: "kuduclusterkey@0"
I20260618 23:28:34.818775 20185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:28:34.848690 20185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:28:34.851228 20185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:28:34.851441 20185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:28:34.851771 20185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:28:34.852188 20185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:28:34.852279 20185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:34.852342 20185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:28:34.852389 20185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:34.860446 20185 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.129:36235
I20260618 23:28:34.860533 20315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.129:36235 every 8 connection(s)
I20260618 23:28:34.861007 20185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
I20260618 23:28:34.863224 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 20185
I20260618 23:28:34.863361 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance
I20260618 23:28:34.863570 8350 external_mini_cluster.cc:1428] Setting key b06e7326c95eb0b1445103eb3b27fa28
Jun 18 23:28:34 dist-test-slave-47qt krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825314, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.39.130@KRBTEST.COM; defaulting to no policy
I20260618 23:28:34.875771 20319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:34.861949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:48099 (local address 127.8.39.190:44023)
0618 23:28:34.865920 (+ 3971us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:34.865922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:34.865929 (+ 7us) server_negotiation.cc:407] Connection header received
0618 23:28:34.865960 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:34.865960 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:34.865970 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:34.866071 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:34.866647 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:34.867162 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:34.868650 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:34.868965 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:34.869055 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:34.872018 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:34.872020 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:34.872020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:34.872028 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:34.873941 (+ 1913us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:34.874455 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:34.874456 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:34.874457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:34.874524 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:34.874846 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:34.874847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:34.874847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:34.875015 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:34.875125 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:34.875548 (+ 423us) server_negotiation.cc:299] Negotiation successful
0618 23:28:34.875641 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3943,"thread_start_us":85,"threads_started":1}
I20260618 23:28:34.876187 20318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:34.861926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:44023 (local address 127.8.39.129:48099)
0618 23:28:34.862104 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:34.862147 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:34.864114 (+ 1967us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:34.866161 (+ 2047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:34.866165 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:34.866350 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:28:34.866540 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:34.866544 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:34.867274 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:34.867275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:34.868533 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:34.868536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:34.869089 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:34.869090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:34.869278 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:34.870125 (+ 847us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:28:34.870135 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:28:34.871902 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:28:34.874039 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:34.874046 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:34.874049 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:34.874355 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:34.874611 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:34.874612 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:34.874612 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:34.874770 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:34.875165 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:28:34.875172 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:28:34.875376 (+ 204us) client_negotiation.cc:769] Sending connection context
0618 23:28:34.875510 (+ 134us) client_negotiation.cc:240] Negotiation successful
0618 23:28:34.875695 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":93,"threads_started":1}
I20260618 23:28:34.876791 20316 heartbeater.cc:347] Connected to a master server at 127.8.39.190:44023
I20260618 23:28:34.876905 20316 heartbeater.cc:464] Registering TS with master...
I20260618 23:28:34.877185 20316 heartbeater.cc:511] Master 127.8.39.190:44023 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.130@KRBTEST.COM" created.
I20260618 23:28:34.877748 20104 ts_manager.cc:194] Registered new tserver with Master: 914fcc9ca1d849f882da228c2e701c4b (127.8.39.129:36235)
I20260618 23:28:34.878734 20104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.129@KRBTEST.COM'} at 127.8.39.129:48099
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
I20260618 23:28:34.894429 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.39.148:42659/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.130:0
--local_ip_for_outbound_sockets=127.8.39.130
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:44023
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 23:28:35 dist-test-slave-47qt krb5kdc[19628](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781825315, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260618 23:28:35.023422 20322 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:28:35.023746 20322 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:28:35.023831 20322 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:28:35.025611 20322 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:28:35.025717 20322 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:28:35.025804 20322 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.130
I20260618 23:28:35.028137 20322 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.39.148:42659/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:44023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20322
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.130
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:28:35.028663 20322 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:28:35.029194 20322 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:28:35.037045 20332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:35.037045 20329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:35.037369 20330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:28:35.037670 20322 server_base.cc:1034] running on GCE node
I20260618 23:28:35.037886 20322 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:28:35.038224 20322 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:28:35.039470 20322 hybrid_clock.cc:648] HybridClock initialized: now 1781825315039446 us; error 34 us; skew 500 ppm
Jun 18 23:28:35 dist-test-slave-47qt krb5kdc[19628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825315, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:28:35.046648 20322 init.cc:377] Logged in from keytab as kudu/127.8.39.130@KRBTEST.COM (short username kudu)
I20260618 23:28:35.047168 20322 webserver.cc:460] Webserver started at http://127.8.39.130:42533/ using document root <none> and password file <none>
I20260618 23:28:35.047442 20322 fs_manager.cc:359] Metadata directory not provided
I20260618 23:28:35.047513 20322 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:28:35.047683 20322 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 23:28:35 dist-test-slave-47qt krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825315, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for HTTP/127.8.39.148@KRBTEST.COM
I20260618 23:28:35.065150 20322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-1/data/instance:
uuid: "917a6854aa7d4bf0a9d3fb402bcded48"
format_stamp: "Formatted at 2026-06-18 23:28:35 on dist-test-slave-47qt"
server_key: "42b4d1cc41f82580f7b55f7a44e9878b"
server_key_iv: "e9dba237b27d1db5370f681fe02da11e"
server_key_version: "kuduclusterkey@0"
I20260618 23:28:35.065766 20322 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance:
uuid: "917a6854aa7d4bf0a9d3fb402bcded48"
format_stamp: "Formatted at 2026-06-18 23:28:35 on dist-test-slave-47qt"
server_key: "42b4d1cc41f82580f7b55f7a44e9878b"
server_key_iv: "e9dba237b27d1db5370f681fe02da11e"
server_key_version: "kuduclusterkey@0"
I20260618 23:28:35.067718 20322 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260618 23:28:35.084324 20339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:35.084554 20322 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260618 23:28:35.084653 20322 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-1/wal
uuid: "917a6854aa7d4bf0a9d3fb402bcded48"
format_stamp: "Formatted at 2026-06-18 23:28:35 on dist-test-slave-47qt"
server_key: "42b4d1cc41f82580f7b55f7a44e9878b"
server_key_iv: "e9dba237b27d1db5370f681fe02da11e"
server_key_version: "kuduclusterkey@0"
I20260618 23:28:35.084781 20322 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:28:35.097245 20322 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:28:35.099679 20322 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:28:35.099853 20322 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:28:35.100111 20322 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:28:35.100499 20322 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:28:35.100553 20322 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:35.100620 20322 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:28:35.100651 20322 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:35.108034 20322 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.130:39241
I20260618 23:28:35.108281 20452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.130:39241 every 8 connection(s)
I20260618 23:28:35.108551 20322 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
I20260618 23:28:35.116102 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 20322
I20260618 23:28:35.116274 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance
I20260618 23:28:35.116506 8350 external_mini_cluster.cc:1428] Setting key 689efbe66bd20faadd9f75506ec3ada1
WARNING: no policy specified for kudu/127.8.39.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.131@KRBTEST.COM" created.
Jun 18 23:28:35 dist-test-slave-47qt krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825315, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:28:35.136637 20319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:35.109629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:52949 (local address 127.8.39.190:44023)
0618 23:28:35.109682 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:35.109684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:35.109915 (+ 231us) server_negotiation.cc:407] Connection header received
0618 23:28:35.110560 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:35.110562 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:35.110578 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:35.110629 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:35.111182 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:35.111724 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:35.112997 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:35.113544 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:35.113741 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:35.131603 (+ 17862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:35.131605 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:35.131606 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:35.131616 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:35.133914 (+ 2298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:35.134970 (+ 1056us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:35.135054 (+ 84us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:35.135055 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:35.135143 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:35.135637 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:35.135638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:35.135638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:35.135804 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:35.135865 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:35.136389 (+ 524us) server_negotiation.cc:299] Negotiation successful
0618 23:28:35.136503 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260618 23:28:35.137521 20455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:35.109359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:44023 (local address 127.8.39.130:52949)
0618 23:28:35.109777 (+ 418us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:35.109828 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:35.110434 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:35.110695 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:35.110699 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:35.110895 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:28:35.111084 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:35.111088 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:35.111827 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:35.111828 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:35.112904 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:35.112906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:35.128476 (+ 15570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:35.128479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:35.128754 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:35.129923 (+ 1169us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:28:35.129936 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:28:35.131467 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:28:35.134048 (+ 2581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:35.134060 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:35.134063 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:35.134798 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:35.135345 (+ 547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:35.135347 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:35.135347 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:35.135555 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:35.135893 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:28:35.135900 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:28:35.136087 (+ 187us) client_negotiation.cc:769] Sending connection context
0618 23:28:35.136199 (+ 112us) client_negotiation.cc:240] Negotiation successful
0618 23:28:35.137269 (+ 1070us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":78,"threads_started":1}
I20260618 23:28:35.138202 20453 heartbeater.cc:347] Connected to a master server at 127.8.39.190:44023
I20260618 23:28:35.138303 20453 heartbeater.cc:464] Registering TS with master...
I20260618 23:28:35.138504 20453 heartbeater.cc:511] Master 127.8.39.190:44023 requested a full tablet report, sending...
I20260618 23:28:35.138901 20104 ts_manager.cc:194] Registered new tserver with Master: 917a6854aa7d4bf0a9d3fb402bcded48 (127.8.39.130:39241)
I20260618 23:28:35.139370 20104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.130@KRBTEST.COM'} at 127.8.39.130:52949
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
I20260618 23:28:35.162755 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.39.148:42659/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.131:0
--local_ip_for_outbound_sockets=127.8.39.131
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:44023
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:28:35.280979 20459 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:28:35.281399 20459 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:28:35.281504 20459 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:28:35.283380 20459 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:28:35.283488 20459 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:28:35.283581 20459 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.131
I20260618 23:28:35.285997 20459 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.39.148:42659/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:44023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20459
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.131
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:28:35.286523 20459 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:28:35.286866 20459 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:28:35.294221 20467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:35.294524 20464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:28:35.295564 20459 server_base.cc:1034] running on GCE node
W20260618 23:28:35.302706 20465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:28:35.303148 20459 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:28:35.303486 20459 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:28:35.310830 20459 hybrid_clock.cc:648] HybridClock initialized: now 1781825315310791 us; error 60 us; skew 500 ppm
Jun 18 23:28:35 dist-test-slave-47qt krb5kdc[19628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825315, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:28:35.315306 20459 init.cc:377] Logged in from keytab as kudu/127.8.39.131@KRBTEST.COM (short username kudu)
I20260618 23:28:35.315996 20459 webserver.cc:460] Webserver started at http://127.8.39.131:41775/ using document root <none> and password file <none>
I20260618 23:28:35.316414 20459 fs_manager.cc:359] Metadata directory not provided
I20260618 23:28:35.316552 20459 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:28:35.316751 20459 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 18 23:28:35 dist-test-slave-47qt krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825315, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for HTTP/127.8.39.148@KRBTEST.COM
I20260618 23:28:35.340065 20459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-2/data/instance:
uuid: "446e7a43656945929deb200748c124ec"
format_stamp: "Formatted at 2026-06-18 23:28:35 on dist-test-slave-47qt"
server_key: "e30e96dca70955954959e3fc53cacdac"
server_key_iv: "b3cc29c32ef3bdb8edb6daa9aeecf66f"
server_key_version: "kuduclusterkey@0"
I20260618 23:28:35.340494 20459 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance:
uuid: "446e7a43656945929deb200748c124ec"
format_stamp: "Formatted at 2026-06-18 23:28:35 on dist-test-slave-47qt"
server_key: "e30e96dca70955954959e3fc53cacdac"
server_key_iv: "b3cc29c32ef3bdb8edb6daa9aeecf66f"
server_key_version: "kuduclusterkey@0"
I20260618 23:28:35.343027 20459 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260618 23:28:35.358130 20474 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:35.358407 20459 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260618 23:28:35.358503 20459 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-2/wal
uuid: "446e7a43656945929deb200748c124ec"
format_stamp: "Formatted at 2026-06-18 23:28:35 on dist-test-slave-47qt"
server_key: "e30e96dca70955954959e3fc53cacdac"
server_key_iv: "b3cc29c32ef3bdb8edb6daa9aeecf66f"
server_key_version: "kuduclusterkey@0"
I20260618 23:28:35.358598 20459 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:28:35.372815 20459 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:28:35.375414 20459 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:28:35.375640 20459 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:28:35.375919 20459 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:28:35.376353 20459 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:28:35.376454 20459 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:35.376528 20459 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:28:35.376725 20459 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:35.388717 20459 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.131:34877
I20260618 23:28:35.389101 20587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.131:34877 every 8 connection(s)
I20260618 23:28:35.389406 20459 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
I20260618 23:28:35.395452 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 20459
I20260618 23:28:35.395550 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance
I20260618 23:28:35.395738 8350 external_mini_cluster.cc:1428] Setting key c924bcf68d237fbf6373c9d679e0e786
Jun 18 23:28:35 dist-test-slave-47qt krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825315, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:28:35.402030 20319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:35.390123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:48095 (local address 127.8.39.190:44023)
0618 23:28:35.390170 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:35.390172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:35.390661 (+ 489us) server_negotiation.cc:407] Connection header received
0618 23:28:35.391485 (+ 824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:35.391486 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:35.391499 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:35.391588 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:35.392310 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:35.393087 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:35.394825 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:35.395144 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:35.395244 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:35.398911 (+ 3667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:35.398913 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:35.398913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:35.398921 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:35.400506 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:35.400990 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:35.400991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:35.400991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:35.401043 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:35.401331 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:35.401331 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:35.401332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:35.401458 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:35.401497 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:35.401855 (+ 358us) server_negotiation.cc:299] Negotiation successful
0618 23:28:35.401936 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260618 23:28:35.402796 20590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:35.390283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:44023 (local address 127.8.39.131:48095)
0618 23:28:35.390543 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:35.390582 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:35.391346 (+ 764us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:35.391665 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:35.391668 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:35.391920 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:28:35.392188 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:35.392194 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:35.393228 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:35.393229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:35.394724 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:35.394727 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:35.395974 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:35.396051 (+ 77us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:35.396196 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:35.397173 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:28:35.397191 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:28:35.398782 (+ 1591us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:28:35.400605 (+ 1823us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:35.400612 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:35.400615 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:35.400905 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:35.401108 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:35.401109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:35.401109 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:35.401259 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:35.401538 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:28:35.401543 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:28:35.401710 (+ 167us) client_negotiation.cc:769] Sending connection context
0618 23:28:35.402387 (+ 677us) client_negotiation.cc:240] Negotiation successful
0618 23:28:35.402550 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":94,"threads_started":1}
I20260618 23:28:35.403560 20588 heartbeater.cc:347] Connected to a master server at 127.8.39.190:44023
I20260618 23:28:35.403654 20588 heartbeater.cc:464] Registering TS with master...
I20260618 23:28:35.403846 20588 heartbeater.cc:511] Master 127.8.39.190:44023 requested a full tablet report, sending...
I20260618 23:28:35.404249 20104 ts_manager.cc:194] Registered new tserver with Master: 446e7a43656945929deb200748c124ec (127.8.39.131:34877)
I20260618 23:28:35.404716 20104 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.131@KRBTEST.COM'} at 127.8.39.131:48095
I20260618 23:28:35.407352 8350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:28:35.407976 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 20185
I20260618 23:28:35.413481 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 20322
I20260618 23:28:35.419057 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 20459
I20260618 23:28:35.424656 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 20072
I20260618 23:28:35.432480 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:44023
--webserver_interface=127.8.39.190
--webserver_port=38583
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:44023
--ranger_config_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.39.148:42659/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.39.148:42659/kms with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:28:35.516054 20591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:28:35.516429 20591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:28:35.516513 20591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:28:35.517843 20591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:28:35.517916 20591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:28:35.517942 20591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:28:35.517972 20591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:28:35.517997 20591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:28:35.519615 20591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.39.148:42659/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:44023
--ranger_config_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:44023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=38583
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:28:35.519901 20591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:28:35.520129 20591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:28:35.525301 20596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:35.525080 20597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:28:35.525254 20591 server_base.cc:1034] running on GCE node
W20260618 23:28:35.525079 20599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:28:35.525985 20591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:28:35.526345 20591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:28:35.527520 20591 hybrid_clock.cc:648] HybridClock initialized: now 1781825315527503 us; error 32 us; skew 500 ppm
Jun 18 23:28:35 dist-test-slave-47qt krb5kdc[19628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825315, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:28:35.530712 20591 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:28:35.531291 20591 webserver.cc:460] Webserver started at http://127.8.39.190:38583/ using document root <none> and password file <none>
I20260618 23:28:35.531553 20591 fs_manager.cc:359] Metadata directory not provided
I20260618 23:28:35.531630 20591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 18 23:28:35 dist-test-slave-47qt krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825315, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for HTTP/127.8.39.148@KRBTEST.COM
I20260618 23:28:35.579435 20591 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:28:35.580201 20607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:35.580368 20591 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:28:35.580461 20591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "1cb91418f59d4ce59dcd70082f5996aa"
format_stamp: "Formatted at 2026-06-18 23:28:33 on dist-test-slave-47qt"
server_key: "fa5da2d2d513ec60a4e769fd2cc152c6"
server_key_iv: "cffd3e7935d30a93941d571e4a2478da"
server_key_version: "kuduclusterkey@0"
I20260618 23:28:35.580806 20591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:28:35.588682 20591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:28:35.591296 20591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:28:35.591472 20591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:28:35.597193 20591 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:44023
I20260618 23:28:35.597211 20659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:44023 every 8 connection(s)
I20260618 23:28:35.597720 20591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:28:35.598250 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 20591
I20260618 23:28:35.598578 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.39.148:42659/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.129:36235
--local_ip_for_outbound_sockets=127.8.39.129
--tserver_master_addrs=127.8.39.190:44023
--webserver_port=34307
--webserver_interface=127.8.39.129
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.39.148:42659/kms with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:28:35.599619 20660 sys_catalog.cc:263] Verifying existing consensus state
I20260618 23:28:35.600873 20660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa: Bootstrap starting.
I20260618 23:28:35.612010 20660 log.cc:824] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa: Log is configured to *not* fsync() on all Append() calls
I20260618 23:28:35.614670 20660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 23:28:35.615031 20660 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa: Bootstrap complete.
I20260618 23:28:35.618431 20660 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb91418f59d4ce59dcd70082f5996aa" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44023 } }
I20260618 23:28:35.618746 20660 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cb91418f59d4ce59dcd70082f5996aa, State: Initialized, Role: FOLLOWER
I20260618 23:28:35.618973 20660 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb91418f59d4ce59dcd70082f5996aa" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44023 } }
I20260618 23:28:35.619045 20660 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:28:35.619082 20660 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:28:35.619127 20660 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [term 1 FOLLOWER]: Advancing to term 2
I20260618 23:28:35.620431 20660 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb91418f59d4ce59dcd70082f5996aa" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44023 } }
I20260618 23:28:35.620540 20660 leader_election.cc:302] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cb91418f59d4ce59dcd70082f5996aa; no voters:
I20260618 23:28:35.620697 20660 leader_election.cc:288] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [CANDIDATE]: Term 2 election: Requested vote from peers
I20260618 23:28:35.620831 20664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [term 2 FOLLOWER]: Leader election won for term 2
I20260618 23:28:35.620968 20660 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:28:35.621089 20664 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [term 2 LEADER]: Becoming Leader. State: Replica: 1cb91418f59d4ce59dcd70082f5996aa, State: Running, Role: LEADER
I20260618 23:28:35.621275 20664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb91418f59d4ce59dcd70082f5996aa" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44023 } }
I20260618 23:28:35.621714 20664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1cb91418f59d4ce59dcd70082f5996aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb91418f59d4ce59dcd70082f5996aa" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44023 } } }
I20260618 23:28:35.621860 20664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [sys.catalog]: This master's current role is: LEADER
I20260618 23:28:35.622025 20664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cb91418f59d4ce59dcd70082f5996aa. Latest consensus state: current_term: 2 leader_uuid: "1cb91418f59d4ce59dcd70082f5996aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cb91418f59d4ce59dcd70082f5996aa" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44023 } } }
I20260618 23:28:35.622117 20664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa [sys.catalog]: This master's current role is: LEADER
I20260618 23:28:35.622428 20660 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260618 23:28:35.622684 20675 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:28:35.625895 20675 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:28:35.626232 20675 catalog_manager.cc:1259] Loaded cluster ID: 16dd47ebf5e44aa9b6d90b72a75d75bb
I20260618 23:28:35.626262 20675 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:28:35.628265 20675 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:28:35.628489 20675 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa: Loaded TSK: 0
I20260618 23:28:35.628808 20675 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260618 23:28:35.724970 20662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:28:35.725267 20662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:28:35.725309 20662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:28:35.727005 20662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:28:35.727114 20662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:28:35.727293 20662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.129
I20260618 23:28:35.729658 20662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.39.148:42659/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.129:36235
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.39.129
--webserver_port=34307
--tserver_master_addrs=127.8.39.190:44023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.129
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:28:35.729899 20662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:28:35.730186 20662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:28:35.736773 20697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:35.736785 20698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:35.737260 20700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:28:35.738116 20662 server_base.cc:1034] running on GCE node
I20260618 23:28:35.738344 20662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:28:35.738610 20662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:28:35.739826 20662 hybrid_clock.cc:648] HybridClock initialized: now 1781825315739805 us; error 32 us; skew 500 ppm
Jun 18 23:28:35 dist-test-slave-47qt krb5kdc[19628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825315, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:28:35.743933 20662 init.cc:377] Logged in from keytab as kudu/127.8.39.129@KRBTEST.COM (short username kudu)
I20260618 23:28:35.744529 20662 webserver.cc:460] Webserver started at http://127.8.39.129:34307/ using document root <none> and password file <none>
I20260618 23:28:35.744863 20662 fs_manager.cc:359] Metadata directory not provided
I20260618 23:28:35.745005 20662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 18 23:28:35 dist-test-slave-47qt krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825315, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for HTTP/127.8.39.148@KRBTEST.COM
I20260618 23:28:35.767410 20662 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:28:35.768941 20708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:35.769181 20662 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:28:35.769289 20662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-0/wal
uuid: "914fcc9ca1d849f882da228c2e701c4b"
format_stamp: "Formatted at 2026-06-18 23:28:34 on dist-test-slave-47qt"
server_key: "9a44590ce3749a9b6e7b29c1110dd002"
server_key_iv: "af754fabd9db7b779e7d7336d1ec7bfa"
server_key_version: "kuduclusterkey@0"
I20260618 23:28:35.769760 20662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:28:35.798128 20662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:28:35.800607 20662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:28:35.800772 20662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:28:35.801069 20662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:28:35.801476 20662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:28:35.801528 20662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:35.801597 20662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:28:35.801637 20662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:35.808115 20662 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.129:36235
I20260618 23:28:35.808642 20662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
I20260618 23:28:35.809533 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 20662
I20260618 23:28:35.809808 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.39.148:42659/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.130:39241
--local_ip_for_outbound_sockets=127.8.39.130
--tserver_master_addrs=127.8.39.190:44023
--webserver_port=42533
--webserver_interface=127.8.39.130
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.39.148:42659/kms with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:28:35.818780 20821 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.129:36235 every 8 connection(s)
Jun 18 23:28:35 dist-test-slave-47qt krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825315, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:28:35.843879 20826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:35.819554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:54881 (local address 127.8.39.190:44023)
0618 23:28:35.825036 (+ 5482us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:35.825048 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:35.829499 (+ 4451us) server_negotiation.cc:407] Connection header received
0618 23:28:35.830370 (+ 871us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:35.830388 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:35.830458 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:35.830586 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:35.831464 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:35.832817 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:35.834787 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:35.835384 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:35.835464 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:35.839537 (+ 4073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:35.839542 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:35.839542 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:35.839556 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:35.841651 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:35.842191 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:35.842194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:35.842194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:35.842300 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:35.842589 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:35.842590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:35.842590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:35.842896 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:35.842994 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:35.843531 (+ 537us) server_negotiation.cc:299] Negotiation successful
0618 23:28:35.843661 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5413,"thread_start_us":93,"threads_started":1}
I20260618 23:28:35.844322 20825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:35.819476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:44023 (local address 127.8.39.129:54881)
0618 23:28:35.829293 (+ 9817us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:35.829364 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:35.830072 (+ 708us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:35.830818 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:35.830821 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:35.831068 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:28:35.831322 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:35.831327 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:35.832944 (+ 1617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:35.832945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:35.834667 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:35.834670 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:35.835609 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:35.835610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:35.835733 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:35.836345 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:28:35.836355 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:28:35.839387 (+ 3032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:28:35.841771 (+ 2384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:35.841779 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:35.841782 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:35.842088 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:35.842381 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:35.842382 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:35.842382 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:35.842508 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:35.843032 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:28:35.843038 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:28:35.843261 (+ 223us) client_negotiation.cc:769] Sending connection context
0618 23:28:35.844008 (+ 747us) client_negotiation.cc:240] Negotiation successful
0618 23:28:35.844151 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9758,"thread_start_us":98,"threads_started":1}
I20260618 23:28:35.845194 20822 heartbeater.cc:347] Connected to a master server at 127.8.39.190:44023
I20260618 23:28:35.845320 20822 heartbeater.cc:464] Registering TS with master...
I20260618 23:28:35.845504 20822 heartbeater.cc:511] Master 127.8.39.190:44023 requested a full tablet report, sending...
W20260618 23:28:35.846371 20822 heartbeater.cc:644] Failed to heartbeat to 127.8.39.190:44023 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260618 23:28:35.846451 20822 heartbeater.cc:464] Registering TS with master...
I20260618 23:28:35.846510 20822 heartbeater.cc:511] Master 127.8.39.190:44023 requested a full tablet report, sending...
I20260618 23:28:35.846853 20822 heartbeater.cc:464] Registering TS with master...
I20260618 23:28:35.846930 20822 heartbeater.cc:511] Master 127.8.39.190:44023 requested a full tablet report, sending...
W20260618 23:28:35.847230 20822 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260618 23:28:35.925453 20824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:28:35.925747 20824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:28:35.925776 20824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:28:35.927301 20824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:28:35.927361 20824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:28:35.927399 20824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.130
I20260618 23:28:35.929066 20824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.39.148:42659/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.130:39241
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.39.130
--webserver_port=42533
--tserver_master_addrs=127.8.39.190:44023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.130
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:28:35.929358 20824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:28:35.929632 20824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:28:35.935892 20832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:35.935894 20831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:35.936350 20834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:28:35.936779 20824 server_base.cc:1034] running on GCE node
I20260618 23:28:35.936995 20824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:28:35.937295 20824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:28:35.938462 20824 hybrid_clock.cc:648] HybridClock initialized: now 1781825315938429 us; error 46 us; skew 500 ppm
Jun 18 23:28:35 dist-test-slave-47qt krb5kdc[19628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825315, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:28:35.942903 20824 init.cc:377] Logged in from keytab as kudu/127.8.39.130@KRBTEST.COM (short username kudu)
I20260618 23:28:35.943470 20824 webserver.cc:460] Webserver started at http://127.8.39.130:42533/ using document root <none> and password file <none>
I20260618 23:28:35.943770 20824 fs_manager.cc:359] Metadata directory not provided
I20260618 23:28:35.943894 20824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 18 23:28:35 dist-test-slave-47qt krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825315, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for HTTP/127.8.39.148@KRBTEST.COM
I20260618 23:28:35.963297 20824 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260618 23:28:35.965420 20842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:35.965672 20824 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260618 23:28:35.965785 20824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-1/wal
uuid: "917a6854aa7d4bf0a9d3fb402bcded48"
format_stamp: "Formatted at 2026-06-18 23:28:35 on dist-test-slave-47qt"
server_key: "42b4d1cc41f82580f7b55f7a44e9878b"
server_key_iv: "e9dba237b27d1db5370f681fe02da11e"
server_key_version: "kuduclusterkey@0"
I20260618 23:28:35.966254 20824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:28:35.997010 20824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:28:35.999616 20824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:28:35.999778 20824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:28:36.000029 20824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:28:36.000377 20824 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:28:36.000424 20824 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:36.000456 20824 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:28:36.000481 20824 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:36.007779 20824 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.130:39241
I20260618 23:28:36.007967 20955 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.130:39241 every 8 connection(s)
I20260618 23:28:36.008287 20824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
Jun 18 23:28:36 dist-test-slave-47qt krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825315, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:28:36.017614 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 20824
I20260618 23:28:36.017908 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.39.148:42659/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.131:34877
--local_ip_for_outbound_sockets=127.8.39.131
--tserver_master_addrs=127.8.39.190:44023
--webserver_port=41775
--webserver_interface=127.8.39.131
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:43989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.39.148:42659/kms with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:28:36.019140 20826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:36.009073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:44629 (local address 127.8.39.190:44023)
0618 23:28:36.009157 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:36.009160 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:36.009430 (+ 270us) server_negotiation.cc:407] Connection header received
0618 23:28:36.009983 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:36.009984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:36.009999 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:36.010055 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:36.011057 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:36.011676 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:36.012961 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:36.013273 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:36.013489 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:36.015880 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:36.015882 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:36.015882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:36.015892 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:36.017607 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:36.018047 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:36.018049 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:36.018049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:36.018108 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:36.018397 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:36.018398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:36.018399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:36.018543 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:36.018587 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:36.018926 (+ 339us) server_negotiation.cc:299] Negotiation successful
0618 23:28:36.019041 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260618 23:28:36.019742 20958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:36.009065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:44023 (local address 127.8.39.130:44629)
0618 23:28:36.009267 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:36.009315 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:36.009848 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:36.010123 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:36.010128 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:36.010446 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:28:36.010821 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:36.010831 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:36.011825 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:36.011827 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:36.012856 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:36.012859 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:36.013521 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:36.013523 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:36.013680 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:36.014267 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:28:36.014278 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:28:36.015768 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:28:36.017712 (+ 1944us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:36.017720 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:36.017722 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:36.017956 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:36.018176 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:36.018177 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:36.018177 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:36.018309 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:36.018627 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:28:36.018632 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:28:36.018750 (+ 118us) client_negotiation.cc:769] Sending connection context
0618 23:28:36.019319 (+ 569us) client_negotiation.cc:240] Negotiation successful
0618 23:28:36.019500 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":113,"threads_started":1}
I20260618 23:28:36.020243 20956 heartbeater.cc:347] Connected to a master server at 127.8.39.190:44023
I20260618 23:28:36.020397 20956 heartbeater.cc:464] Registering TS with master...
I20260618 23:28:36.020646 20956 heartbeater.cc:511] Master 127.8.39.190:44023 requested a full tablet report, sending...
W20260618 23:28:36.021381 20956 heartbeater.cc:644] Failed to heartbeat to 127.8.39.190:44023 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260618 23:28:36.021500 20956 heartbeater.cc:464] Registering TS with master...
I20260618 23:28:36.021607 20956 heartbeater.cc:511] Master 127.8.39.190:44023 requested a full tablet report, sending...
I20260618 23:28:36.022115 20956 heartbeater.cc:464] Registering TS with master...
I20260618 23:28:36.022233 20956 heartbeater.cc:511] Master 127.8.39.190:44023 requested a full tablet report, sending...
W20260618 23:28:36.022581 20956 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260618 23:28:36.127874 20959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:28:36.128165 20959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:28:36.128201 20959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:28:36.129951 20959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:28:36.130019 20959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:28:36.130060 20959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.131
I20260618 23:28:36.132400 20959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:43989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.39.148:42659/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.131:34877
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.39.131
--webserver_port=41775
--tserver_master_addrs=127.8.39.190:44023
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20959
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.131
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:28:36.132653 20959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:28:36.132911 20959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:28:36.140348 20965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:36.140348 20967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:36.140350 20964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:28:36.141954 20959 server_base.cc:1034] running on GCE node
I20260618 23:28:36.142175 20959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:28:36.142457 20959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:28:36.146548 20959 hybrid_clock.cc:648] HybridClock initialized: now 1781825316146508 us; error 45 us; skew 500 ppm
Jun 18 23:28:36 dist-test-slave-47qt krb5kdc[19628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825316, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:28:36.152053 20959 init.cc:377] Logged in from keytab as kudu/127.8.39.131@KRBTEST.COM (short username kudu)
I20260618 23:28:36.152594 20959 webserver.cc:460] Webserver started at http://127.8.39.131:41775/ using document root <none> and password file <none>
I20260618 23:28:36.152873 20959 fs_manager.cc:359] Metadata directory not provided
I20260618 23:28:36.152943 20959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 18 23:28:36 dist-test-slave-47qt krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825316, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for HTTP/127.8.39.148@KRBTEST.COM
I20260618 23:28:36.173638 20959 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260618 23:28:36.187103 20976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:36.187381 20959 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.001s sys 0.000s
I20260618 23:28:36.187477 20959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-2/wal
uuid: "446e7a43656945929deb200748c124ec"
format_stamp: "Formatted at 2026-06-18 23:28:35 on dist-test-slave-47qt"
server_key: "e30e96dca70955954959e3fc53cacdac"
server_key_iv: "b3cc29c32ef3bdb8edb6daa9aeecf66f"
server_key_version: "kuduclusterkey@0"
I20260618 23:28:36.187925 20959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:28:36.215077 20959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:28:36.217425 20959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:28:36.217614 20959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:28:36.217895 20959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:28:36.218269 20959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:28:36.218323 20959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:36.218359 20959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:28:36.218386 20959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:36.225775 20959 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.131:34877
I20260618 23:28:36.226241 20959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
I20260618 23:28:36.226981 21089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.131:34877 every 8 connection(s)
I20260618 23:28:36.228804 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 20959
Jun 18 23:28:36 dist-test-slave-47qt krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825316, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:28:36.238752 20826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:36.227960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:53019 (local address 127.8.39.190:44023)
0618 23:28:36.228010 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:36.228012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:36.228026 (+ 14us) server_negotiation.cc:407] Connection header received
0618 23:28:36.228694 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:36.228695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:36.228713 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:36.228788 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:36.229570 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:36.230340 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:36.232254 (+ 1914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:36.232565 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:36.232648 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:36.235399 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:36.235400 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:36.235401 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:36.235408 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:36.237175 (+ 1767us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:36.237666 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:36.237667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:36.237667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:36.237723 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:36.238016 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:36.238017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:36.238017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:36.238143 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:36.238182 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:36.238547 (+ 365us) server_negotiation.cc:299] Negotiation successful
0618 23:28:36.238656 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260618 23:28:36.239646 21092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:36.227617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:44023 (local address 127.8.39.131:53019)
0618 23:28:36.227854 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:36.227906 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:36.228538 (+ 632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:36.228860 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:36.228864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:36.229142 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:28:36.229434 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:36.229441 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:36.230818 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:36.230819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:36.232140 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:36.232143 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:36.232735 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:36.232736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:36.232884 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:36.233545 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:28:36.233556 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:28:36.235283 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:28:36.237264 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:36.237272 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:36.237275 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:36.237579 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:36.237789 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:36.237790 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:36.237790 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:36.237920 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:36.238224 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:28:36.238229 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:28:36.238390 (+ 161us) client_negotiation.cc:769] Sending connection context
0618 23:28:36.239164 (+ 774us) client_negotiation.cc:240] Negotiation successful
0618 23:28:36.239360 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":121,"threads_started":1}
I20260618 23:28:36.240569 21090 heartbeater.cc:347] Connected to a master server at 127.8.39.190:44023
I20260618 23:28:36.240710 21090 heartbeater.cc:464] Registering TS with master...
I20260618 23:28:36.240960 21090 heartbeater.cc:511] Master 127.8.39.190:44023 requested a full tablet report, sending...
I20260618 23:28:36.241469 21094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:36.230430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58774 (local address 127.8.39.190:44023)
0618 23:28:36.232502 (+ 2072us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:36.232504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:36.232511 (+ 7us) server_negotiation.cc:407] Connection header received
0618 23:28:36.232533 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:36.232533 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:36.232545 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:36.232609 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:36.233450 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:36.234178 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:36.236052 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:36.236373 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:36.236440 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:36.237883 (+ 1443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:36.237885 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:36.237885 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:36.237892 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:36.239716 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:36.240165 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:36.240166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:36.240166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:36.240227 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:36.240519 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:36.240520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:36.240521 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:36.240654 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:36.240812 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:36.241193 (+ 381us) server_negotiation.cc:299] Negotiation successful
0618 23:28:36.241350 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2043,"thread_start_us":271,"threads_started":1}
W20260618 23:28:36.241627 21090 heartbeater.cc:644] Failed to heartbeat to 127.8.39.190:44023 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260618 23:28:36.241681 21090 heartbeater.cc:464] Registering TS with master...
I20260618 23:28:36.241724 21090 heartbeater.cc:511] Master 127.8.39.190:44023 requested a full tablet report, sending...
I20260618 23:28:36.242084 21090 heartbeater.cc:464] Registering TS with master...
I20260618 23:28:36.242138 21090 heartbeater.cc:511] Master 127.8.39.190:44023 requested a full tablet report, sending...
W20260618 23:28:36.242470 21090 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 18 23:28:36 dist-test-slave-47qt krb5kdc[19628](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781825316, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:28:36.847425 20822 heartbeater.cc:464] Registering TS with master...
I20260618 23:28:36.847632 20822 heartbeater.cc:511] Master 127.8.39.190:44023 requested a full tablet report, sending...
I20260618 23:28:36.848012 20624 ts_manager.cc:194] Registered new tserver with Master: 914fcc9ca1d849f882da228c2e701c4b (127.8.39.129:36235)
I20260618 23:28:36.848692 20624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.129@KRBTEST.COM'} at 127.8.39.129:54881
I20260618 23:28:37.022807 20956 heartbeater.cc:464] Registering TS with master...
I20260618 23:28:37.022974 20956 heartbeater.cc:511] Master 127.8.39.190:44023 requested a full tablet report, sending...
I20260618 23:28:37.023295 20624 ts_manager.cc:194] Registered new tserver with Master: 917a6854aa7d4bf0a9d3fb402bcded48 (127.8.39.130:39241)
I20260618 23:28:37.023814 20624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.130@KRBTEST.COM'} at 127.8.39.130:44629
Jun 18 23:28:37 dist-test-slave-47qt krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825316, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for HTTP/127.8.39.148@KRBTEST.COM
I20260618 23:28:37.242655 21090 heartbeater.cc:464] Registering TS with master...
I20260618 23:28:37.242790 21090 heartbeater.cc:511] Master 127.8.39.190:44023 requested a full tablet report, sending...
I20260618 23:28:37.243136 20624 ts_manager.cc:194] Registered new tserver with Master: 446e7a43656945929deb200748c124ec (127.8.39.131:34877)
I20260618 23:28:37.243628 20624 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.131@KRBTEST.COM'} at 127.8.39.131:53019
I20260618 23:28:37.245549 8350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:28:37.257782 21111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:37.249401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58776 (local address 127.8.39.190:44023)
0618 23:28:37.249541 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:37.249544 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:37.249645 (+ 101us) server_negotiation.cc:407] Connection header received
0618 23:28:37.249692 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:37.249693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:37.249706 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:37.249789 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:37.250480 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:37.251185 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:37.252416 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:37.252743 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:37.252827 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:37.254366 (+ 1539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:37.254368 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:37.254368 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:37.254376 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:37.256153 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:37.256551 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:37.256552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:37.256553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:37.256618 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:37.256825 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:37.256825 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:37.256826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:37.256966 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:37.257069 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:37.257470 (+ 401us) server_negotiation.cc:299] Negotiation successful
0618 23:28:37.257639 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
I20260618 23:28:37.260353 20624 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58776:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:28:37.262599 20624 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:28:37.275589 21117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:37.265456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:39241 (local address 127.0.0.1:51918)
0618 23:28:37.265907 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:37.265946 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:37.266012 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:37.269229 (+ 3217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:37.269230 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:37.269232 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:28:37.269292 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:37.269293 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:37.270621 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:37.270622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:37.272848 (+ 2226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:37.272853 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:37.275278 (+ 2425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:37.275279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:37.275377 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:37.275394 (+ 17us) client_negotiation.cc:769] Sending connection context
0618 23:28:37.275443 (+ 49us) client_negotiation.cc:240] Negotiation successful
0618 23:28:37.275498 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"mutex_wait_us":436,"thread_start_us":92,"threads_started":1}
I20260618 23:28:37.276273 21122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:37.265879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51918 (local address 127.8.39.130:39241)
0618 23:28:37.269023 (+ 3144us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:37.269027 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:37.269036 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:28:37.269065 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:37.269068 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:37.269081 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:37.269358 (+ 277us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:28:37.269409 (+ 51us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:37.270528 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:37.274379 (+ 3851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:37.275152 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:37.276016 (+ 864us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:37.276117 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:37.276144 (+ 27us) server_negotiation.cc:299] Negotiation successful
0618 23:28:37.276195 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3114,"thread_start_us":60,"threads_started":1}
I20260618 23:28:37.276512 21119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:37.265460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:34877 (local address 127.0.0.1:42670)
0618 23:28:37.268664 (+ 3204us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:37.268674 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:37.268728 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:37.268953 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:37.268957 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:37.268960 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:28:37.269045 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:37.269047 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:37.270537 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:37.270538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:37.274977 (+ 4439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:37.274980 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:37.276287 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:37.276287 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:37.276347 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:37.276351 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:28:37.276393 (+ 42us) client_negotiation.cc:240] Negotiation successful
0618 23:28:37.276432 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3186,"thread_start_us":88,"threads_started":1}
I20260618 23:28:37.276837 21120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:37.265737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42670 (local address 127.8.39.131:34877)
0618 23:28:37.268611 (+ 2874us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:37.268615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:37.268781 (+ 166us) server_negotiation.cc:407] Connection header received
0618 23:28:37.268818 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:37.268820 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:37.268836 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:37.268910 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:28:37.269190 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:37.270436 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:37.275083 (+ 4647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:37.276113 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:37.276169 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:37.276223 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:37.276714 (+ 491us) server_negotiation.cc:299] Negotiation successful
0618 23:28:37.276761 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2846,"thread_start_us":67,"threads_started":1}
I20260618 23:28:37.278215 21118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:37.265693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:36235 (local address 127.0.0.1:58378)
0618 23:28:37.270869 (+ 5176us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:37.270879 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:37.270959 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:37.271191 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:37.271192 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:37.271194 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:28:37.271285 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:37.271287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:37.272711 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:37.272712 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:37.274263 (+ 1551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:37.274267 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:37.277954 (+ 3687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:37.277955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:37.278032 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:37.278036 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:28:37.278088 (+ 52us) client_negotiation.cc:240] Negotiation successful
0618 23:28:37.278130 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5160,"mutex_wait_us":34,"thread_start_us":61,"threads_started":1}
I20260618 23:28:37.278267 21121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:37.265790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58378 (local address 127.8.39.129:36235)
0618 23:28:37.268545 (+ 2755us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:37.268550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:37.270933 (+ 2383us) server_negotiation.cc:407] Connection header received
0618 23:28:37.271035 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:37.271037 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:37.271051 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:37.271101 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:28:37.271359 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:37.272533 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:37.276843 (+ 4310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:37.277645 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:37.277694 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:37.277744 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:37.278135 (+ 391us) server_negotiation.cc:299] Negotiation successful
0618 23:28:37.278184 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2722,"thread_start_us":47,"threads_started":1}
I20260618 23:28:37.279181 20738 tablet_service.cc:1467] Processing CreateTablet for tablet 14831e9a262e45369949a93aeb225042 (DEFAULT_TABLE table=test-table [id=52a6ade129864c7baac2253215d95331]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:28:37.279642 20738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14831e9a262e45369949a93aeb225042. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:28:37.280296 20872 tablet_service.cc:1467] Processing CreateTablet for tablet 14831e9a262e45369949a93aeb225042 (DEFAULT_TABLE table=test-table [id=52a6ade129864c7baac2253215d95331]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:28:37.280629 20872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14831e9a262e45369949a93aeb225042. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:28:37.284122 21024 tablet_service.cc:1467] Processing CreateTablet for tablet 14831e9a262e45369949a93aeb225042 (DEFAULT_TABLE table=test-table [id=52a6ade129864c7baac2253215d95331]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:28:37.284478 21024 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14831e9a262e45369949a93aeb225042. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:28:37.285279 21124 tablet_bootstrap.cc:492] T 14831e9a262e45369949a93aeb225042 P 917a6854aa7d4bf0a9d3fb402bcded48: Bootstrap starting.
I20260618 23:28:37.286001 21124 tablet_bootstrap.cc:654] T 14831e9a262e45369949a93aeb225042 P 917a6854aa7d4bf0a9d3fb402bcded48: Neither blocks nor log segments found. Creating new log.
I20260618 23:28:37.286357 21124 log.cc:824] T 14831e9a262e45369949a93aeb225042 P 917a6854aa7d4bf0a9d3fb402bcded48: Log is configured to *not* fsync() on all Append() calls
I20260618 23:28:37.286747 21123 tablet_bootstrap.cc:492] T 14831e9a262e45369949a93aeb225042 P 914fcc9ca1d849f882da228c2e701c4b: Bootstrap starting.
I20260618 23:28:37.288064 21123 tablet_bootstrap.cc:654] T 14831e9a262e45369949a93aeb225042 P 914fcc9ca1d849f882da228c2e701c4b: Neither blocks nor log segments found. Creating new log.
I20260618 23:28:37.288069 21126 tablet_bootstrap.cc:492] T 14831e9a262e45369949a93aeb225042 P 446e7a43656945929deb200748c124ec: Bootstrap starting.
I20260618 23:28:37.288460 21123 log.cc:824] T 14831e9a262e45369949a93aeb225042 P 914fcc9ca1d849f882da228c2e701c4b: Log is configured to *not* fsync() on all Append() calls
I20260618 23:28:37.288671 21126 tablet_bootstrap.cc:654] T 14831e9a262e45369949a93aeb225042 P 446e7a43656945929deb200748c124ec: Neither blocks nor log segments found. Creating new log.
I20260618 23:28:37.288970 21126 log.cc:824] T 14831e9a262e45369949a93aeb225042 P 446e7a43656945929deb200748c124ec: Log is configured to *not* fsync() on all Append() calls
I20260618 23:28:37.289719 21126 tablet_bootstrap.cc:492] T 14831e9a262e45369949a93aeb225042 P 446e7a43656945929deb200748c124ec: No bootstrap required, opened a new log
I20260618 23:28:37.289719 21123 tablet_bootstrap.cc:492] T 14831e9a262e45369949a93aeb225042 P 914fcc9ca1d849f882da228c2e701c4b: No bootstrap required, opened a new log
I20260618 23:28:37.289786 21123 ts_tablet_manager.cc:1397] T 14831e9a262e45369949a93aeb225042 P 914fcc9ca1d849f882da228c2e701c4b: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260618 23:28:37.289786 21126 ts_tablet_manager.cc:1397] T 14831e9a262e45369949a93aeb225042 P 446e7a43656945929deb200748c124ec: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260618 23:28:37.292217 21123 raft_consensus.cc:348] T 14831e9a262e45369949a93aeb225042 P 914fcc9ca1d849f882da228c2e701c4b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917a6854aa7d4bf0a9d3fb402bcded48" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 39241 } } peers { permanent_uuid: "446e7a43656945929deb200748c124ec" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 34877 } } peers { permanent_uuid: "914fcc9ca1d849f882da228c2e701c4b" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 36235 } }
I20260618 23:28:37.292383 21123 raft_consensus.cc:374] T 14831e9a262e45369949a93aeb225042 P 914fcc9ca1d849f882da228c2e701c4b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:28:37.292420 21123 raft_consensus.cc:729] T 14831e9a262e45369949a93aeb225042 P 914fcc9ca1d849f882da228c2e701c4b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 914fcc9ca1d849f882da228c2e701c4b, State: Initialized, Role: FOLLOWER
I20260618 23:28:37.292431 21126 raft_consensus.cc:348] T 14831e9a262e45369949a93aeb225042 P 446e7a43656945929deb200748c124ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917a6854aa7d4bf0a9d3fb402bcded48" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 39241 } } peers { permanent_uuid: "446e7a43656945929deb200748c124ec" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 34877 } } peers { permanent_uuid: "914fcc9ca1d849f882da228c2e701c4b" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 36235 } }
I20260618 23:28:37.292555 21126 raft_consensus.cc:374] T 14831e9a262e45369949a93aeb225042 P 446e7a43656945929deb200748c124ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:28:37.292598 21126 raft_consensus.cc:729] T 14831e9a262e45369949a93aeb225042 P 446e7a43656945929deb200748c124ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 446e7a43656945929deb200748c124ec, State: Initialized, Role: FOLLOWER
I20260618 23:28:37.292712 21126 consensus_queue.cc:260] T 14831e9a262e45369949a93aeb225042 P 446e7a43656945929deb200748c124ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917a6854aa7d4bf0a9d3fb402bcded48" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 39241 } } peers { permanent_uuid: "446e7a43656945929deb200748c124ec" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 34877 } } peers { permanent_uuid: "914fcc9ca1d849f882da228c2e701c4b" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 36235 } }
I20260618 23:28:37.292855 21123 consensus_queue.cc:260] T 14831e9a262e45369949a93aeb225042 P 914fcc9ca1d849f882da228c2e701c4b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917a6854aa7d4bf0a9d3fb402bcded48" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 39241 } } peers { permanent_uuid: "446e7a43656945929deb200748c124ec" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 34877 } } peers { permanent_uuid: "914fcc9ca1d849f882da228c2e701c4b" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 36235 } }
I20260618 23:28:37.292989 21126 ts_tablet_manager.cc:1428] T 14831e9a262e45369949a93aeb225042 P 446e7a43656945929deb200748c124ec: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260618 23:28:37.293165 21090 heartbeater.cc:503] Master 127.8.39.190:44023 was elected leader, sending a full tablet report...
I20260618 23:28:37.293378 21123 ts_tablet_manager.cc:1428] T 14831e9a262e45369949a93aeb225042 P 914fcc9ca1d849f882da228c2e701c4b: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260618 23:28:37.293534 20822 heartbeater.cc:503] Master 127.8.39.190:44023 was elected leader, sending a full tablet report...
I20260618 23:28:37.296221 21124 tablet_bootstrap.cc:492] T 14831e9a262e45369949a93aeb225042 P 917a6854aa7d4bf0a9d3fb402bcded48: No bootstrap required, opened a new log
I20260618 23:28:37.296308 21124 ts_tablet_manager.cc:1397] T 14831e9a262e45369949a93aeb225042 P 917a6854aa7d4bf0a9d3fb402bcded48: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.000s
I20260618 23:28:37.298045 21124 raft_consensus.cc:348] T 14831e9a262e45369949a93aeb225042 P 917a6854aa7d4bf0a9d3fb402bcded48 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917a6854aa7d4bf0a9d3fb402bcded48" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 39241 } } peers { permanent_uuid: "446e7a43656945929deb200748c124ec" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 34877 } } peers { permanent_uuid: "914fcc9ca1d849f882da228c2e701c4b" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 36235 } }
I20260618 23:28:37.298202 21124 raft_consensus.cc:374] T 14831e9a262e45369949a93aeb225042 P 917a6854aa7d4bf0a9d3fb402bcded48 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:28:37.298245 21124 raft_consensus.cc:729] T 14831e9a262e45369949a93aeb225042 P 917a6854aa7d4bf0a9d3fb402bcded48 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 917a6854aa7d4bf0a9d3fb402bcded48, State: Initialized, Role: FOLLOWER
I20260618 23:28:37.298476 21124 consensus_queue.cc:260] T 14831e9a262e45369949a93aeb225042 P 917a6854aa7d4bf0a9d3fb402bcded48 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917a6854aa7d4bf0a9d3fb402bcded48" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 39241 } } peers { permanent_uuid: "446e7a43656945929deb200748c124ec" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 34877 } } peers { permanent_uuid: "914fcc9ca1d849f882da228c2e701c4b" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 36235 } }
I20260618 23:28:37.298817 20956 heartbeater.cc:503] Master 127.8.39.190:44023 was elected leader, sending a full tablet report...
I20260618 23:28:37.299062 21124 ts_tablet_manager.cc:1428] T 14831e9a262e45369949a93aeb225042 P 917a6854aa7d4bf0a9d3fb402bcded48: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260618 23:28:37.319823 20823 tablet.cc:2369] T 14831e9a262e45369949a93aeb225042 P 914fcc9ca1d849f882da228c2e701c4b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:28:37.409070 21131 raft_consensus.cc:482] T 14831e9a262e45369949a93aeb225042 P 917a6854aa7d4bf0a9d3fb402bcded48 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:28:37.409263 21131 raft_consensus.cc:504] T 14831e9a262e45369949a93aeb225042 P 917a6854aa7d4bf0a9d3fb402bcded48 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917a6854aa7d4bf0a9d3fb402bcded48" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 39241 } } peers { permanent_uuid: "446e7a43656945929deb200748c124ec" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 34877 } } peers { permanent_uuid: "914fcc9ca1d849f882da228c2e701c4b" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 36235 } }
I20260618 23:28:37.409678 21131 leader_election.cc:288] T 14831e9a262e45369949a93aeb225042 P 917a6854aa7d4bf0a9d3fb402bcded48 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 446e7a43656945929deb200748c124ec (127.8.39.131:34877), 914fcc9ca1d849f882da228c2e701c4b (127.8.39.129:36235)
I20260618 23:28:37.414642 21121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:37.410151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:52189 (local address 127.8.39.129:36235)
0618 23:28:37.410214 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:37.410216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:37.410656 (+ 440us) server_negotiation.cc:407] Connection header received
0618 23:28:37.410767 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:37.410768 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:37.410780 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:37.410809 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:28:37.411081 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:37.411787 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:37.413680 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:37.414249 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:37.414301 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:37.414311 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:37.414526 (+ 215us) server_negotiation.cc:299] Negotiation successful
0618 23:28:37.414568 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260618 23:28:37.414917 21132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:37.409936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:36235 (local address 127.8.39.130:52189)
0618 23:28:37.410586 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:37.410598 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:37.410685 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:37.410876 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:37.410877 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:37.410879 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:28:37.410992 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:37.410995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:37.412338 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:37.412338 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:37.413568 (+ 1230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:37.413571 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:37.414371 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:37.414372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:37.414443 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:37.414446 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:28:37.414773 (+ 327us) client_negotiation.cc:240] Negotiation successful
0618 23:28:37.414821 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":104,"threads_started":1}
I20260618 23:28:37.415256 20763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14831e9a262e45369949a93aeb225042" candidate_uuid: "917a6854aa7d4bf0a9d3fb402bcded48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "914fcc9ca1d849f882da228c2e701c4b" is_pre_election: true
I20260618 23:28:37.415431 20763 raft_consensus.cc:2454] T 14831e9a262e45369949a93aeb225042 P 914fcc9ca1d849f882da228c2e701c4b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 917a6854aa7d4bf0a9d3fb402bcded48 in term 0.
I20260618 23:28:37.415669 20843 leader_election.cc:302] T 14831e9a262e45369949a93aeb225042 P 917a6854aa7d4bf0a9d3fb402bcded48 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 914fcc9ca1d849f882da228c2e701c4b, 917a6854aa7d4bf0a9d3fb402bcded48; no voters:
I20260618 23:28:37.415804 21131 raft_consensus.cc:2781] T 14831e9a262e45369949a93aeb225042 P 917a6854aa7d4bf0a9d3fb402bcded48 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:28:37.415854 21131 raft_consensus.cc:482] T 14831e9a262e45369949a93aeb225042 P 917a6854aa7d4bf0a9d3fb402bcded48 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:28:37.415884 21131 raft_consensus.cc:3037] T 14831e9a262e45369949a93aeb225042 P 917a6854aa7d4bf0a9d3fb402bcded48 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:28:37.416797 21131 raft_consensus.cc:504] T 14831e9a262e45369949a93aeb225042 P 917a6854aa7d4bf0a9d3fb402bcded48 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917a6854aa7d4bf0a9d3fb402bcded48" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 39241 } } peers { permanent_uuid: "446e7a43656945929deb200748c124ec" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 34877 } } peers { permanent_uuid: "914fcc9ca1d849f882da228c2e701c4b" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 36235 } }
I20260618 23:28:37.416940 21131 leader_election.cc:288] T 14831e9a262e45369949a93aeb225042 P 917a6854aa7d4bf0a9d3fb402bcded48 [CANDIDATE]: Term 1 election: Requested vote from peers 446e7a43656945929deb200748c124ec (127.8.39.131:34877), 914fcc9ca1d849f882da228c2e701c4b (127.8.39.129:36235)
I20260618 23:28:37.417167 20763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14831e9a262e45369949a93aeb225042" candidate_uuid: "917a6854aa7d4bf0a9d3fb402bcded48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "914fcc9ca1d849f882da228c2e701c4b"
I20260618 23:28:37.417285 20763 raft_consensus.cc:3037] T 14831e9a262e45369949a93aeb225042 P 914fcc9ca1d849f882da228c2e701c4b [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:28:37.417915 20763 raft_consensus.cc:2454] T 14831e9a262e45369949a93aeb225042 P 914fcc9ca1d849f882da228c2e701c4b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 917a6854aa7d4bf0a9d3fb402bcded48 in term 1.
I20260618 23:28:37.418177 20843 leader_election.cc:302] T 14831e9a262e45369949a93aeb225042 P 917a6854aa7d4bf0a9d3fb402bcded48 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 914fcc9ca1d849f882da228c2e701c4b, 917a6854aa7d4bf0a9d3fb402bcded48; no voters:
I20260618 23:28:37.418262 21131 raft_consensus.cc:2781] T 14831e9a262e45369949a93aeb225042 P 917a6854aa7d4bf0a9d3fb402bcded48 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:28:37.418370 21131 raft_consensus.cc:686] T 14831e9a262e45369949a93aeb225042 P 917a6854aa7d4bf0a9d3fb402bcded48 [term 1 LEADER]: Becoming Leader. State: Replica: 917a6854aa7d4bf0a9d3fb402bcded48, State: Running, Role: LEADER
I20260618 23:28:37.418445 21131 consensus_queue.cc:237] T 14831e9a262e45369949a93aeb225042 P 917a6854aa7d4bf0a9d3fb402bcded48 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917a6854aa7d4bf0a9d3fb402bcded48" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 39241 } } peers { permanent_uuid: "446e7a43656945929deb200748c124ec" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 34877 } } peers { permanent_uuid: "914fcc9ca1d849f882da228c2e701c4b" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 36235 } }
I20260618 23:28:37.419085 21120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:37.410490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:48975 (local address 127.8.39.131:34877)
0618 23:28:37.410531 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:37.410533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:37.410708 (+ 175us) server_negotiation.cc:407] Connection header received
0618 23:28:37.410823 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:37.410824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:37.410836 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:37.410863 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:28:37.411064 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:37.411768 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:37.418074 (+ 6306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:37.418656 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:37.418749 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:37.418761 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:37.418970 (+ 209us) server_negotiation.cc:299] Negotiation successful
0618 23:28:37.419020 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260618 23:28:37.419276 21133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:37.410300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:34877 (local address 127.8.39.130:48975)
0618 23:28:37.410620 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:37.410633 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:37.410751 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:37.410922 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:37.410922 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:37.410924 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:28:37.410998 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:37.411000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:37.412807 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:37.412808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:37.417946 (+ 5138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:37.417949 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:37.418792 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:37.418793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:37.418877 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:37.418881 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:28:37.419163 (+ 282us) client_negotiation.cc:240] Negotiation successful
0618 23:28:37.419204 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":2,"thread_start_us":82,"threads_started":1}
I20260618 23:28:37.419225 20624 catalog_manager.cc:5495] T 14831e9a262e45369949a93aeb225042 P 917a6854aa7d4bf0a9d3fb402bcded48 reported cstate change: term changed from 0 to 1, leader changed from <none> to 917a6854aa7d4bf0a9d3fb402bcded48 (127.8.39.130). New cstate: current_term: 1 leader_uuid: "917a6854aa7d4bf0a9d3fb402bcded48" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "917a6854aa7d4bf0a9d3fb402bcded48" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 39241 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "446e7a43656945929deb200748c124ec" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 34877 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "914fcc9ca1d849f882da228c2e701c4b" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 36235 } health_report { overall_health: UNKNOWN } } }
I20260618 23:28:37.419914 21044 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14831e9a262e45369949a93aeb225042" candidate_uuid: "917a6854aa7d4bf0a9d3fb402bcded48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446e7a43656945929deb200748c124ec" is_pre_election: true
I20260618 23:28:37.420059 21044 raft_consensus.cc:2454] T 14831e9a262e45369949a93aeb225042 P 446e7a43656945929deb200748c124ec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 917a6854aa7d4bf0a9d3fb402bcded48 in term 0.
I20260618 23:28:37.420207 21043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14831e9a262e45369949a93aeb225042" candidate_uuid: "917a6854aa7d4bf0a9d3fb402bcded48" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "446e7a43656945929deb200748c124ec"
I20260618 23:28:37.420295 21043 raft_consensus.cc:3037] T 14831e9a262e45369949a93aeb225042 P 446e7a43656945929deb200748c124ec [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:28:37.421326 21043 raft_consensus.cc:2454] T 14831e9a262e45369949a93aeb225042 P 446e7a43656945929deb200748c124ec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 917a6854aa7d4bf0a9d3fb402bcded48 in term 1.
Jun 18 23:28:37 dist-test-slave-47qt krb5kdc[19628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825316, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for HTTP/127.8.39.148@KRBTEST.COM
I20260618 23:28:37.454077 21122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:37.450669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51920 (local address 127.8.39.130:39241)
0618 23:28:37.450892 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:37.450894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:37.450903 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:28:37.450945 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:37.450946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:37.450971 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:37.451047 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:28:37.451236 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:37.451941 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:37.452974 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:37.453296 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:37.453354 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:37.453546 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:28:37.453685 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:28:37.453755 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:37.453977 (+ 222us) server_negotiation.cc:299] Negotiation successful
0618 23:28:37.453998 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196}
I20260618 23:28:37.456350 20763 raft_consensus.cc:1261] T 14831e9a262e45369949a93aeb225042 P 914fcc9ca1d849f882da228c2e701c4b [term 1 FOLLOWER]: Refusing update from remote peer 917a6854aa7d4bf0a9d3fb402bcded48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:28:37.456786 21135 consensus_queue.cc:1035] T 14831e9a262e45369949a93aeb225042 P 917a6854aa7d4bf0a9d3fb402bcded48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "914fcc9ca1d849f882da228c2e701c4b" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 36235 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:28:37.459213 21043 raft_consensus.cc:1261] T 14831e9a262e45369949a93aeb225042 P 446e7a43656945929deb200748c124ec [term 1 FOLLOWER]: Refusing update from remote peer 917a6854aa7d4bf0a9d3fb402bcded48: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:28:37.459612 21135 consensus_queue.cc:1035] T 14831e9a262e45369949a93aeb225042 P 917a6854aa7d4bf0a9d3fb402bcded48 [LEADER]: Connected to new peer: Peer: permanent_uuid: "446e7a43656945929deb200748c124ec" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 34877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:28:37.466765 21138 mvcc.cc:205] Tried to move back new op lower bound from 7298356500297388032 to 7298356500145840128. Current Snapshot: MvccSnapshot[applied={T|T < 7298356500297388032}]
I20260618 23:28:37.466953 21141 mvcc.cc:205] Tried to move back new op lower bound from 7298356500297388032 to 7298356500145840128. Current Snapshot: MvccSnapshot[applied={T|T < 7298356500297388032}]
I20260618 23:28:37.469436 20624 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:28:37.469545 20624 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58776:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:28:37.470402 20624 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1cb91418f59d4ce59dcd70082f5996aa: Sending DeleteTablet for 3 replicas of tablet 14831e9a262e45369949a93aeb225042
I20260618 23:28:37.471012 20738 tablet_service.cc:1514] Processing DeleteTablet for tablet 14831e9a262e45369949a93aeb225042 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:28:37 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:58378
I20260618 23:28:37.471258 21024 tablet_service.cc:1514] Processing DeleteTablet for tablet 14831e9a262e45369949a93aeb225042 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:28:37 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:42670
I20260618 23:28:37.471673 20872 tablet_service.cc:1514] Processing DeleteTablet for tablet 14831e9a262e45369949a93aeb225042 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:28:37 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:51918
I20260618 23:28:37.472062 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 20662
I20260618 23:28:37.483899 21147 tablet_replica.cc:330] T 14831e9a262e45369949a93aeb225042 P 446e7a43656945929deb200748c124ec: stopping tablet replica
I20260618 23:28:37.484076 21147 raft_consensus.cc:2229] T 14831e9a262e45369949a93aeb225042 P 446e7a43656945929deb200748c124ec [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:28:37.484241 21147 raft_consensus.cc:2258] T 14831e9a262e45369949a93aeb225042 P 446e7a43656945929deb200748c124ec [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:28:37.484422 21148 tablet_replica.cc:330] T 14831e9a262e45369949a93aeb225042 P 917a6854aa7d4bf0a9d3fb402bcded48: stopping tablet replica
I20260618 23:28:37.484530 21148 raft_consensus.cc:2229] T 14831e9a262e45369949a93aeb225042 P 917a6854aa7d4bf0a9d3fb402bcded48 [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:28:37.484632 21148 raft_consensus.cc:2258] T 14831e9a262e45369949a93aeb225042 P 917a6854aa7d4bf0a9d3fb402bcded48 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:28:37.485307 21148 ts_tablet_manager.cc:1905] T 14831e9a262e45369949a93aeb225042 P 917a6854aa7d4bf0a9d3fb402bcded48: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:28:37.485390 21147 ts_tablet_manager.cc:1905] T 14831e9a262e45369949a93aeb225042 P 446e7a43656945929deb200748c124ec: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260618 23:28:37.487272 20843 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.39.129:36235 (error 108)
I20260618 23:28:37.487402 21147 ts_tablet_manager.cc:1918] T 14831e9a262e45369949a93aeb225042 P 446e7a43656945929deb200748c124ec: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:28:37.487470 21147 log.cc:1196] T 14831e9a262e45369949a93aeb225042 P 446e7a43656945929deb200748c124ec: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-2/wal/wals/14831e9a262e45369949a93aeb225042
I20260618 23:28:37.487671 21148 ts_tablet_manager.cc:1918] T 14831e9a262e45369949a93aeb225042 P 917a6854aa7d4bf0a9d3fb402bcded48: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:28:37.487742 21147 ts_tablet_manager.cc:1939] T 14831e9a262e45369949a93aeb225042 P 446e7a43656945929deb200748c124ec: Deleting consensus metadata
I20260618 23:28:37.487892 21148 log.cc:1196] T 14831e9a262e45369949a93aeb225042 P 917a6854aa7d4bf0a9d3fb402bcded48: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781825053542692-8350-0/minicluster-data/ts-1/wal/wals/14831e9a262e45369949a93aeb225042
I20260618 23:28:37.488171 20610 catalog_manager.cc:4841] TS 446e7a43656945929deb200748c124ec (127.8.39.131:34877): tablet 14831e9a262e45369949a93aeb225042 (table test-table [id=52a6ade129864c7baac2253215d95331]) successfully deleted
I20260618 23:28:37.488205 21148 ts_tablet_manager.cc:1939] T 14831e9a262e45369949a93aeb225042 P 917a6854aa7d4bf0a9d3fb402bcded48: Deleting consensus metadata
W20260618 23:28:37.488431 20608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.39.129:36235 (error 108)
I20260618 23:28:37.489208 20608 catalog_manager.cc:4841] TS 917a6854aa7d4bf0a9d3fb402bcded48 (127.8.39.130:39241): tablet 14831e9a262e45369949a93aeb225042 (table test-table [id=52a6ade129864c7baac2253215d95331]) successfully deleted
I20260618 23:28:37.489444 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 20824
I20260618 23:28:37.491978 21118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:37.491848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:36235 (local address 127.0.0.1:58390)
0618 23:28:37.491900 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:37.491941 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.129:36235: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260618 23:28:37.493876 20608 catalog_manager.cc:4584] TS 914fcc9ca1d849f882da228c2e701c4b (127.8.39.129:36235): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 14831e9a262e45369949a93aeb225042: Network error: Client connection negotiation failed: client connection to 127.8.39.129:36235: connect: Connection refused (error 111)
I20260618 23:28:37.500458 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 20959
I20260618 23:28:37.506661 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 20591
I20260618 23:28:37.515535 8350 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260618 23:28:37.860411 8350 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260618 23:28:37.860509 8350 mini_ranger.cc:66] Stopping Ranger...
I20260618 23:28:38.205659 8350 mini_ranger.cc:68] Stopped Ranger
2026-06-18 23:28:38.205 UTC [19658] LOG: received smart shutdown request
2026-06-18 23:28:38.209 UTC [19658] LOG: background worker "logical replication launcher" (PID 19667) exited with exit code 1
2026-06-18 23:28:38.210 UTC [19662] LOG: shutting down
2026-06-18 23:28:38.411 UTC [19658] LOG: database system is shut down
2026-06-18T23:28:38Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49251 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:28:38 dist-test-slave-47qt krb5kdc[21160](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:28:38 dist-test-slave-47qt krb5kdc[21160](info): set up 2 sockets
Jun 18 23:28:38 dist-test-slave-47qt krb5kdc[21160](info): commencing operation
krb5kdc: starting...
W20260618 23:28:39.588982 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:28:39 dist-test-slave-47qt krb5kdc[21160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:28:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:28:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.39.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
I20260618 23:28:39.717953 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:33745
--webserver_interface=127.8.39.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:42419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:33745,127.8.39.189:45893,127.8.39.188:33187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:28:39.804188 21173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:28:39.804450 21173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:28:39.804490 21173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:28:39.805701 21173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:28:39.805781 21173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:28:39.805807 21173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:28:39.805828 21173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:28:39.805859 21173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:28:39.807420 21173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:42419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:33745,127.8.39.189:45893,127.8.39.188:33187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:33745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:28:39.807731 21173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:28:39.807969 21173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:28:39.813033 21179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:28:39.813160 21173 server_base.cc:1034] running on GCE node
W20260618 23:28:39.813292 21181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:39.813297 21178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:28:39.813583 21173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:28:39.813928 21173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:28:39.815074 21173 hybrid_clock.cc:648] HybridClock initialized: now 1781825319815064 us; error 30 us; skew 500 ppm
Jun 18 23:28:39 dist-test-slave-47qt krb5kdc[21160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825319, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:28:39.817425 21173 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:28:39.817938 21173 webserver.cc:460] Webserver started at http://127.8.39.190:37623/ using document root <none> and password file <none>
I20260618 23:28:39.818216 21173 fs_manager.cc:359] Metadata directory not provided
I20260618 23:28:39.818291 21173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:28:39.818437 21173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:28:39.820003 21173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "6a9879e09006498eacf678c8304ead84"
format_stamp: "Formatted at 2026-06-18 23:28:39 on dist-test-slave-47qt"
server_key: "02151854192d3405d07e158efe90134b"
server_key_iv: "67996e3d32a518947dd2e46759b3d854"
server_key_version: "encryptionkey@0"
I20260618 23:28:39.820456 21173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "6a9879e09006498eacf678c8304ead84"
format_stamp: "Formatted at 2026-06-18 23:28:39 on dist-test-slave-47qt"
server_key: "02151854192d3405d07e158efe90134b"
server_key_iv: "67996e3d32a518947dd2e46759b3d854"
server_key_version: "encryptionkey@0"
I20260618 23:28:39.823643 21173 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260618 23:28:39.824939 21188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:39.825275 21173 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:28:39.825402 21173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "6a9879e09006498eacf678c8304ead84"
format_stamp: "Formatted at 2026-06-18 23:28:39 on dist-test-slave-47qt"
server_key: "02151854192d3405d07e158efe90134b"
server_key_iv: "67996e3d32a518947dd2e46759b3d854"
server_key_version: "encryptionkey@0"
I20260618 23:28:39.825503 21173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:28:39.832613 21173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:28:39.834928 21173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:28:39.835094 21173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:28:39.839196 21173 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:33745
I20260618 23:28:39.839236 21240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:33745 every 8 connection(s)
I20260618 23:28:39.839581 21173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:28:39.840492 21241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:28:39.842113 21241 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33745 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:28:39.844309 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 21173
I20260618 23:28:39.844424 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:28:39.844597 8350 external_mini_cluster.cc:1428] Setting key 283f327e33071e2ffa543fa4d4ba3961
Jun 18 23:28:39 dist-test-slave-47qt krb5kdc[21160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825319, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.39.189@KRBTEST.COM; defaulting to no policy
I20260618 23:28:39.853950 21244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:39.842994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41128 (local address 127.8.39.190:33745)
0618 23:28:39.843183 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:39.843190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:39.843302 (+ 112us) server_negotiation.cc:407] Connection header received
0618 23:28:39.843881 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:39.843883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:39.843900 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:39.843981 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:39.844534 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:39.845862 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:39.847802 (+ 1940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:39.848117 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:39.848249 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:39.850680 (+ 2431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:39.850687 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:39.850688 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:39.850701 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:39.852307 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:39.852702 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:39.852705 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:39.852705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:39.852809 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:39.853018 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:39.853020 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:39.853020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:39.853308 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:39.853404 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:39.853580 (+ 176us) server_negotiation.cc:299] Negotiation successful
0618 23:28:39.853709 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":86,"threads_started":1}
I20260618 23:28:39.853951 21243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:39.842948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:33745 (local address 127.0.0.1:41128)
0618 23:28:39.843092 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:39.843139 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:39.843703 (+ 564us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:39.844002 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:39.844004 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:39.844247 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:28:39.844410 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:39.844414 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:39.846111 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:39.846114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:39.847689 (+ 1575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:39.847693 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:39.848255 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:39.848256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:39.848354 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:39.848898 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:28:39.848908 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:28:39.850560 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:28:39.852415 (+ 1855us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:39.852422 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:39.852424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:39.852620 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:39.852863 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:39.852864 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:39.852865 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:39.852953 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:39.853381 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:28:39.853386 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:28:39.853444 (+ 58us) client_negotiation.cc:769] Sending connection context
0618 23:28:39.853554 (+ 110us) client_negotiation.cc:240] Negotiation successful
0618 23:28:39.853766 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":70,"thread_start_us":68,"threads_started":1}
I20260618 23:28:39.854743 21241 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.39.189" port: 45893 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:28:39.855106 21243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:39.854988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:45893 (local address 127.0.0.1:56678)
0618 23:28:39.855041 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:39.855084 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.189:45893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260618 23:28:39.855247 21190 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.39.189:45893: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.189@KRBTEST.COM" created.
I20260618 23:28:39.855697 21243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:39.855615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:45893 (local address 127.0.0.1:56682)
0618 23:28:39.855657 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:39.855681 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.189:45893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260618 23:28:39.855839 21241 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.39.189:45893: Network error: Client connection negotiation failed: client connection to 127.8.39.189:45893: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.189.keytab.
Entry for principal kudu/127.8.39.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.189.keytab.
I20260618 23:28:39.872426 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.189:45893
--webserver_interface=127.8.39.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.189.keytab
--principal=kudu/127.8.39.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:42419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:33745,127.8.39.189:45893,127.8.39.188:33187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:28:39.905160 21241 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.39.189" port: 45893 } attempt: 1
I20260618 23:28:39.905719 21243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:39.905579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:45893 (local address 127.0.0.1:56694)
0618 23:28:39.905648 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:39.905691 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.189:45893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260618 23:28:39.906478 21243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:39.906348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:45893 (local address 127.0.0.1:56696)
0618 23:28:39.906421 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:39.906456 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.189:45893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260618 23:28:39.906620 21241 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.39.189:45893: Network error: Client connection negotiation failed: client connection to 127.8.39.189:45893: connect: Connection refused (error 111)
W20260618 23:28:39.966355 21248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:28:39.966708 21248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:28:39.966748 21248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:28:39.967900 21248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:28:39.967978 21248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:28:39.968003 21248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:28:39.968030 21248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:28:39.968060 21248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:28:39.969617 21248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:42419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:33745,127.8.39.189:45893,127.8.39.188:33187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.189.keytab
--principal=kudu/127.8.39.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.189:45893
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.39.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21248
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:28:39.969906 21248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:28:39.970140 21248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 23:28:39.974886 21241 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.39.189" port: 45893 } attempt: 2
W20260618 23:28:39.974933 21253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:39.975042 21254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:39.975121 21256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:28:39.975322 21243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:39.975158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:45893 (local address 127.0.0.1:56702)
0618 23:28:39.975261 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:39.975294 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.189:45893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260618 23:28:39.975711 21248 server_base.cc:1034] running on GCE node
I20260618 23:28:39.975867 21243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:39.975720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:45893 (local address 127.0.0.1:56710)
0618 23:28:39.975794 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:39.975823 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.189:45893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260618 23:28:39.975989 21241 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.39.189:45893: Network error: Client connection negotiation failed: client connection to 127.8.39.189:45893: connect: Connection refused (error 111)
I20260618 23:28:39.976014 21248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:28:39.976346 21248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:28:39.977509 21248 hybrid_clock.cc:648] HybridClock initialized: now 1781825319977501 us; error 39 us; skew 500 ppm
Jun 18 23:28:39 dist-test-slave-47qt krb5kdc[21160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825319, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:28:39.980484 21248 init.cc:377] Logged in from keytab as kudu/127.8.39.189@KRBTEST.COM (short username kudu)
I20260618 23:28:39.981361 21248 webserver.cc:460] Webserver started at http://127.8.39.189:34027/ using document root <none> and password file <none>
I20260618 23:28:39.981629 21248 fs_manager.cc:359] Metadata directory not provided
I20260618 23:28:39.981701 21248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:28:39.981845 21248 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:28:39.982928 21248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-1/data/instance:
uuid: "d76bcc45c09d4dd8931756897d54e4a3"
format_stamp: "Formatted at 2026-06-18 23:28:39 on dist-test-slave-47qt"
server_key: "29f2f69754ffcdd1c3070d2187316ea1"
server_key_iv: "333947711c4762bbc50d40990a978c6f"
server_key_version: "encryptionkey@0"
I20260618 23:28:39.983481 21248 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-1/wal/instance:
uuid: "d76bcc45c09d4dd8931756897d54e4a3"
format_stamp: "Formatted at 2026-06-18 23:28:39 on dist-test-slave-47qt"
server_key: "29f2f69754ffcdd1c3070d2187316ea1"
server_key_iv: "333947711c4762bbc50d40990a978c6f"
server_key_version: "encryptionkey@0"
I20260618 23:28:39.985057 21248 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:28:39.987046 21263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:39.987381 21248 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.002s
I20260618 23:28:39.987519 21248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-1/wal
uuid: "d76bcc45c09d4dd8931756897d54e4a3"
format_stamp: "Formatted at 2026-06-18 23:28:39 on dist-test-slave-47qt"
server_key: "29f2f69754ffcdd1c3070d2187316ea1"
server_key_iv: "333947711c4762bbc50d40990a978c6f"
server_key_version: "encryptionkey@0"
I20260618 23:28:39.987607 21248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:28:40.001019 21248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:28:40.003347 21248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:28:40.003479 21248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:28:40.009434 21248 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.189:45893
I20260618 23:28:40.009464 21315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.189:45893 every 8 connection(s)
I20260618 23:28:40.009819 21248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-1/data/info.pb
I20260618 23:28:40.010669 21316 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:28:40.012151 21316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33745 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 23:28:40 dist-test-slave-47qt krb5kdc[21160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825319, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.189@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:28:40.018347 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 21248
I20260618 23:28:40.018422 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-1/wal/instance
I20260618 23:28:40.018589 8350 external_mini_cluster.cc:1428] Setting key 03d8dcbd7ed5e7fbe92d270bad1b448b
I20260618 23:28:40.022758 21244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.012902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41142 (local address 127.8.39.190:33745)
0618 23:28:40.012970 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:40.012972 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:40.013203 (+ 231us) server_negotiation.cc:407] Connection header received
0618 23:28:40.013682 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:40.013684 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:40.013695 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:40.013785 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:40.014514 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.015004 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.016016 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.016259 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.016325 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:40.018718 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:40.018720 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:40.018720 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:40.018734 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:40.021222 (+ 2488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:40.021687 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:40.021689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:40.021689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:40.021754 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:40.022026 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:40.022027 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:40.022028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:40.022154 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:40.022233 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:40.022576 (+ 343us) server_negotiation.cc:299] Negotiation successful
0618 23:28:40.022660 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260618 23:28:40.022960 21318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.012805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:33745 (local address 127.0.0.1:41142)
0618 23:28:40.012972 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:40.013014 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:40.013548 (+ 534us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:40.013896 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:40.013899 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:40.014167 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:28:40.014364 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:40.014367 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.015129 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.015129 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:40.015924 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:40.015926 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.016357 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.016358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:40.016467 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:40.016970 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:28:40.016978 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:28:40.018595 (+ 1617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:28:40.021336 (+ 2741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:40.021345 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:40.021347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:40.021588 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:40.021832 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:40.021833 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:40.021833 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:40.021943 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:40.022225 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:28:40.022232 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:28:40.022425 (+ 193us) client_negotiation.cc:769] Sending connection context
0618 23:28:40.022557 (+ 132us) client_negotiation.cc:240] Negotiation successful
0618 23:28:40.022723 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
I20260618 23:28:40.023564 21316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.39.189" port: 45893 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.39.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.188@KRBTEST.COM" created.
Jun 18 23:28:40 dist-test-slave-47qt krb5kdc[21160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825319, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.189@KRBTEST.COM for kudu/127.8.39.189@KRBTEST.COM
I20260618 23:28:40.034425 21318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.023831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:45893 (local address 127.0.0.1:56726)
0618 23:28:40.023879 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:40.023890 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:40.023928 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:40.024207 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:40.024208 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:40.024380 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:28:40.024432 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:40.024433 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.025692 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.025693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:40.026357 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:40.026363 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.027081 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.027082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:40.027156 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:40.027560 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:28:40.027565 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:28:40.030416 (+ 2851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:28:40.032628 (+ 2212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:40.032631 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:40.032632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:40.033202 (+ 570us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:40.033499 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:40.033500 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:40.033500 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:40.033559 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:40.033904 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:28:40.033905 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:28:40.033942 (+ 37us) client_negotiation.cc:769] Sending connection context
0618 23:28:40.033989 (+ 47us) client_negotiation.cc:240] Negotiation successful
0618 23:28:40.034297 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260618 23:28:40.034880 21320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.023927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56726 (local address 127.8.39.189:45893)
0618 23:28:40.024065 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:40.024069 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:40.024085 (+ 16us) server_negotiation.cc:407] Connection header received
0618 23:28:40.024104 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:40.024106 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:40.024125 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:40.024207 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:40.024510 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.025565 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.026615 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.026994 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.027082 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:40.030517 (+ 3435us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:40.030525 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:40.030525 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:40.030531 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:40.032419 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:40.033344 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:40.033347 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:40.033347 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:40.033425 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:40.033625 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:40.033625 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:40.033626 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:40.033819 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:40.034636 (+ 817us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:40.034689 (+ 53us) server_negotiation.cc:299] Negotiation successful
0618 23:28:40.034774 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260618 23:28:40.035899 21316 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.39.188" port: 33187 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:28:40.036329 21318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.036207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:33187 (local address 127.0.0.1:55358)
0618 23:28:40.036257 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:40.036304 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:33187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260618 23:28:40.036473 21264 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.39.188:33187: connect: Connection refused (error 111)
I20260618 23:28:40.036953 21318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.036860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:33187 (local address 127.0.0.1:55360)
0618 23:28:40.036888 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:40.036912 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:33187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260618 23:28:40.037032 21316 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.39.188:33187: Network error: Client connection negotiation failed: client connection to 127.8.39.188:33187: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.188.keytab.
Entry for principal kudu/127.8.39.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.188.keytab.
I20260618 23:28:40.054133 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.188:33187
--webserver_interface=127.8.39.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.188.keytab
--principal=kudu/127.8.39.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:42419
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:33745,127.8.39.189:45893,127.8.39.188:33187
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:28:40.067152 21241 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.39.189" port: 45893 } attempt: 3
Jun 18 23:28:40 dist-test-slave-47qt krb5kdc[21160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825319, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for kudu/127.8.39.189@KRBTEST.COM
I20260618 23:28:40.080102 21243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.067548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:45893 (local address 127.0.0.1:56734)
0618 23:28:40.067623 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:40.067638 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:40.067707 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:40.067898 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:40.067899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:40.068089 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:28:40.068169 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:40.068171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.069282 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.069283 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:40.070135 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:40.070138 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.070763 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.070764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:40.070870 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:40.074708 (+ 3838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:28:40.074714 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:28:40.076500 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:28:40.078398 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:40.078400 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:40.078401 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:40.078652 (+ 251us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:40.078876 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:40.078876 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:40.078877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:40.078951 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:40.079630 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:28:40.079631 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:28:40.079694 (+ 63us) client_negotiation.cc:769] Sending connection context
0618 23:28:40.079820 (+ 126us) client_negotiation.cc:240] Negotiation successful
0618 23:28:40.079964 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260618 23:28:40.080530 21320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.067684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56734 (local address 127.8.39.189:45893)
0618 23:28:40.067731 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:40.067733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:40.067746 (+ 13us) server_negotiation.cc:407] Connection header received
0618 23:28:40.067778 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:40.067779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:40.067802 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:40.067875 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:40.068258 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.069186 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.070233 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.070609 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.070691 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:40.076592 (+ 5901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:40.076593 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:40.076594 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:40.076602 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:40.078308 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:40.078732 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:40.078733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:40.078734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:40.078790 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:40.079040 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:40.079041 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:40.079041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:40.079203 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:40.079483 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:40.080311 (+ 828us) server_negotiation.cc:299] Negotiation successful
0618 23:28:40.080403 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260618 23:28:40.080899 21241 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.39.188" port: 33187 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:28:40.081382 21243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.081175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:33187 (local address 127.0.0.1:55362)
0618 23:28:40.081316 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:40.081347 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:33187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
I20260618 23:28:40.081727 21243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.081577 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:33187 (local address 127.0.0.1:55378)
0618 23:28:40.081672 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:40.081697 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:33187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260618 23:28:40.081820 21241 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.39.188:33187: Network error: Client connection negotiation failed: client connection to 127.8.39.188:33187: connect: Connection refused (error 111)
I20260618 23:28:40.086259 21316 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.39.188" port: 33187 } attempt: 1
I20260618 23:28:40.086669 21318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.086544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:33187 (local address 127.0.0.1:55384)
0618 23:28:40.086603 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:40.086643 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:33187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260618 23:28:40.087013 21318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.086925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:33187 (local address 127.0.0.1:55394)
0618 23:28:40.086969 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:40.086995 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:33187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260618 23:28:40.087142 21316 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.39.188:33187: Network error: Client connection negotiation failed: client connection to 127.8.39.188:33187: connect: Connection refused (error 111)
I20260618 23:28:40.112988 21241 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.39.188" port: 33187 } attempt: 1
I20260618 23:28:40.113494 21243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.113362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:33187 (local address 127.0.0.1:55400)
0618 23:28:40.113426 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:40.113467 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:33187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260618 23:28:40.113991 21243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.113863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:33187 (local address 127.0.0.1:55416)
0618 23:28:40.113940 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:40.113971 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:33187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260618 23:28:40.114176 21241 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.39.188:33187: Network error: Client connection negotiation failed: client connection to 127.8.39.188:33187: connect: Connection refused (error 111)
W20260618 23:28:40.149605 21323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:28:40.149806 21323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:28:40.149829 21323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:28:40.150960 21323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:28:40.151005 21323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:28:40.151018 21323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:28:40.151031 21323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:28:40.151043 21323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:28:40.152511 21323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:42419
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:33745,127.8.39.189:45893,127.8.39.188:33187
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.188.keytab
--principal=kudu/127.8.39.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.188:33187
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.39.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:28:40.152763 21323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:28:40.152998 21323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 23:28:40.155354 21316 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.39.188" port: 33187 } attempt: 2
I20260618 23:28:40.156139 21318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.155814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:33187 (local address 127.0.0.1:55428)
0618 23:28:40.156009 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:40.156069 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:33187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260618 23:28:40.156667 21318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.156533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:33187 (local address 127.0.0.1:55436)
0618 23:28:40.156598 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:40.156635 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:33187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260618 23:28:40.156836 21316 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.39.188:33187: Network error: Client connection negotiation failed: client connection to 127.8.39.188:33187: connect: Connection refused (error 111)
W20260618 23:28:40.158445 21330 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:40.158447 21329 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:40.158589 21332 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:28:40.159350 21323 server_base.cc:1034] running on GCE node
I20260618 23:28:40.159580 21323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:28:40.160115 21323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:28:40.161369 21323 hybrid_clock.cc:648] HybridClock initialized: now 1781825320161298 us; error 73 us; skew 500 ppm
Jun 18 23:28:40 dist-test-slave-47qt krb5kdc[21160](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825320, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:28:40.164943 21323 init.cc:377] Logged in from keytab as kudu/127.8.39.188@KRBTEST.COM (short username kudu)
I20260618 23:28:40.165971 21323 webserver.cc:460] Webserver started at http://127.8.39.188:43107/ using document root <none> and password file <none>
I20260618 23:28:40.166682 21323 fs_manager.cc:359] Metadata directory not provided
I20260618 23:28:40.166814 21323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:28:40.167011 21323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:28:40.168602 21323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-2/data/instance:
uuid: "1cdc7121b4774eb2abaf02f91cd1ce1c"
format_stamp: "Formatted at 2026-06-18 23:28:40 on dist-test-slave-47qt"
server_key: "98db3dd7670b938e1bdea365483266dc"
server_key_iv: "6850a71e0bb7cf94f2fae0db4c7695f6"
server_key_version: "encryptionkey@0"
I20260618 23:28:40.169077 21323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-2/wal/instance:
uuid: "1cdc7121b4774eb2abaf02f91cd1ce1c"
format_stamp: "Formatted at 2026-06-18 23:28:40 on dist-test-slave-47qt"
server_key: "98db3dd7670b938e1bdea365483266dc"
server_key_iv: "6850a71e0bb7cf94f2fae0db4c7695f6"
server_key_version: "encryptionkey@0"
I20260618 23:28:40.170603 21323 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:28:40.171545 21339 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:40.171871 21323 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:28:40.171996 21323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-2/wal
uuid: "1cdc7121b4774eb2abaf02f91cd1ce1c"
format_stamp: "Formatted at 2026-06-18 23:28:40 on dist-test-slave-47qt"
server_key: "98db3dd7670b938e1bdea365483266dc"
server_key_iv: "6850a71e0bb7cf94f2fae0db4c7695f6"
server_key_version: "encryptionkey@0"
I20260618 23:28:40.172091 21323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:28:40.189385 21241 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.39.188" port: 33187 } attempt: 2
I20260618 23:28:40.189873 21243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.189723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:33187 (local address 127.0.0.1:55442)
0618 23:28:40.189800 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:40.189843 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:33187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260618 23:28:40.190585 21243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.190464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:33187 (local address 127.0.0.1:55454)
0618 23:28:40.190527 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:40.190560 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:33187: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260618 23:28:40.190770 21241 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.39.188:33187: Network error: Client connection negotiation failed: client connection to 127.8.39.188:33187: connect: Connection refused (error 111)
I20260618 23:28:40.191330 21323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:28:40.193743 21323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:28:40.193917 21323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:28:40.197731 21323 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.188:33187
I20260618 23:28:40.197798 21391 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.188:33187 every 8 connection(s)
I20260618 23:28:40.198146 21323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-2/data/info.pb
I20260618 23:28:40.199055 21392 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:28:40.200582 21392 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33745 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:28:40.205101 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 21323
I20260618 23:28:40.205252 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781825053542692-8350-0/minicluster-data/master-2/wal/instance
I20260618 23:28:40.205418 8350 external_mini_cluster.cc:1428] Setting key b2f117fd4d21b9a431f4894f62184cf6
Jun 18 23:28:40 dist-test-slave-47qt krb5kdc[21160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825320, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.188@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:28:40.210273 21244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.201382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41156 (local address 127.8.39.190:33745)
0618 23:28:40.201466 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:40.201468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:40.201633 (+ 165us) server_negotiation.cc:407] Connection header received
0618 23:28:40.202049 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:40.202051 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:40.202061 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:40.202127 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:40.202711 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.203196 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.204539 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.204722 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.204784 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:40.207213 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:40.207215 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:40.207216 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:40.207224 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:40.208794 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:40.209222 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:40.209224 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:40.209225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:40.209378 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:40.209648 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:40.209649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:40.209649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:40.209761 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:40.209818 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:40.210122 (+ 304us) server_negotiation.cc:299] Negotiation successful
0618 23:28:40.210203 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260618 23:28:40.210515 21394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.201316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:33745 (local address 127.0.0.1:41156)
0618 23:28:40.201482 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:40.201522 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:40.201960 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:40.202180 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:40.202183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:40.202414 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:28:40.202593 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:40.202597 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.203398 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.203400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:40.204442 (+ 1042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:40.204447 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.204803 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.204804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:40.204905 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:40.205517 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:28:40.205530 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:28:40.207080 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:28:40.208887 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:40.208892 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:40.208894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:40.209104 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:40.209450 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:40.209451 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:40.209451 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:40.209571 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:40.209830 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:28:40.209834 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:28:40.209986 (+ 152us) client_negotiation.cc:769] Sending connection context
0618 23:28:40.210068 (+ 82us) client_negotiation.cc:240] Negotiation successful
0618 23:28:40.210233 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260618 23:28:40.211066 21392 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.39.189" port: 45893 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 23:28:40 dist-test-slave-47qt krb5kdc[21160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.188@KRBTEST.COM
Jun 18 23:28:40 dist-test-slave-47qt krb5kdc[21160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825320, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.188@KRBTEST.COM for kudu/127.8.39.189@KRBTEST.COM
I20260618 23:28:40.215696 21396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.205833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55468 (local address 127.8.39.188:33187)
0618 23:28:40.205955 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:40.205959 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:40.206080 (+ 121us) server_negotiation.cc:407] Connection header received
0618 23:28:40.206126 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:40.206129 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:40.206153 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:40.206232 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:40.207456 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.208329 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.209441 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.209849 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.209912 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:40.212507 (+ 2595us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:40.212515 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:40.212516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:40.212522 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:40.214515 (+ 1993us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:40.214836 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:40.214839 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:40.214840 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:40.214899 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:40.215119 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:40.215120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:40.215120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:40.215289 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:40.215432 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:40.215518 (+ 86us) server_negotiation.cc:299] Negotiation successful
0618 23:28:40.215573 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":63,"threads_started":1}
I20260618 23:28:40.218340 21394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.211321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:45893 (local address 127.0.0.1:56736)
0618 23:28:40.211381 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:40.211393 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:40.211441 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:40.211583 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:40.211584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:40.211717 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:28:40.211757 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:40.211758 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.212419 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.212421 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:40.213045 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:40.213048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.213461 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.213462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:40.213530 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:40.213972 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:28:40.213976 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:28:40.215261 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:28:40.217292 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:40.217293 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:40.217293 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:40.217459 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:40.217695 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:40.217695 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:40.217696 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:40.217755 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:40.218006 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:28:40.218007 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:28:40.218049 (+ 42us) client_negotiation.cc:769] Sending connection context
0618 23:28:40.218152 (+ 103us) client_negotiation.cc:240] Negotiation successful
0618 23:28:40.218240 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260618 23:28:40.218354 21320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.211408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56736 (local address 127.8.39.189:45893)
0618 23:28:40.211449 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:40.211451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:40.211461 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:28:40.211501 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:40.211506 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:40.211515 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:40.211578 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:40.211837 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.212345 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.213159 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.213373 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.213461 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:40.215394 (+ 1933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:40.215395 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:40.215395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:40.215402 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:40.217185 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:40.217561 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:40.217563 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:40.217563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:40.217626 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:40.217824 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:40.217825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:40.217825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:40.217940 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:40.218015 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:40.218184 (+ 169us) server_negotiation.cc:299] Negotiation successful
0618 23:28:40.218270 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260618 23:28:40.218943 21392 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.39.188" port: 33187 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 23:28:40 dist-test-slave-47qt krb5kdc[21160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
Jun 18 23:28:40 dist-test-slave-47qt krb5kdc[21160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825320, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.188@KRBTEST.COM for kudu/127.8.39.188@KRBTEST.COM
I20260618 23:28:40.224768 21244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.216743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41158 (local address 127.8.39.190:33745)
0618 23:28:40.216780 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:40.216782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:40.216792 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:28:40.216816 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:40.216817 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:40.216824 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:40.216876 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:40.217467 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.217922 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.218997 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.219298 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.219388 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:40.221916 (+ 2528us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:40.221918 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:40.221918 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:40.221925 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:40.223513 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:40.223951 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:40.223952 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:40.223952 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:40.224013 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:40.224233 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:40.224234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:40.224234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:40.224338 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:40.224495 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:40.224571 (+ 76us) server_negotiation.cc:299] Negotiation successful
0618 23:28:40.224648 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260618 23:28:40.226447 21394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.219199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:33187 (local address 127.0.0.1:55476)
0618 23:28:40.219245 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:40.219254 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:40.219287 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:40.219416 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:40.219417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:40.219572 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:28:40.219603 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:40.219605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.220556 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.220557 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:40.221016 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:40.221017 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.221715 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.221716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:40.221759 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:40.222181 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:28:40.222187 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:28:40.223717 (+ 1530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:28:40.225470 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:40.225471 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:40.225472 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:40.225629 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:40.225843 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:40.225844 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:40.225844 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:40.225896 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:40.226141 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:28:40.226142 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:28:40.226178 (+ 36us) client_negotiation.cc:769] Sending connection context
0618 23:28:40.226246 (+ 68us) client_negotiation.cc:240] Negotiation successful
0618 23:28:40.226339 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260618 23:28:40.226500 21396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.219262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55476 (local address 127.8.39.188:33187)
0618 23:28:40.219301 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:40.219303 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:40.219311 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:28:40.219332 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:40.219333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:40.219345 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:40.219413 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:40.219698 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.220230 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.221155 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.221605 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.221691 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:40.223802 (+ 2111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:40.223803 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:40.223803 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:40.223810 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:40.225379 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:40.225720 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:40.225721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:40.225722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:40.225770 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:40.225969 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:40.225970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:40.225970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:40.226075 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:40.226150 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:40.226276 (+ 126us) server_negotiation.cc:299] Negotiation successful
0618 23:28:40.226389 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":22}
I20260618 23:28:40.228018 21392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c: Bootstrap starting.
I20260618 23:28:40.228878 21392 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c: Neither blocks nor log segments found. Creating new log.
I20260618 23:28:40.229220 21392 log.cc:824] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c: Log is configured to *not* fsync() on all Append() calls
Jun 18 23:28:40 dist-test-slave-47qt krb5kdc[21160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825319, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.189@KRBTEST.COM
I20260618 23:28:40.230137 21392 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c: No bootstrap required, opened a new log
I20260618 23:28:40.231905 21392 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9879e09006498eacf678c8304ead84" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33745 } } peers { permanent_uuid: "d76bcc45c09d4dd8931756897d54e4a3" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 45893 } } peers { permanent_uuid: "1cdc7121b4774eb2abaf02f91cd1ce1c" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 33187 } }
I20260618 23:28:40.232052 21392 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:28:40.232106 21392 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cdc7121b4774eb2abaf02f91cd1ce1c, State: Initialized, Role: FOLLOWER
I20260618 23:28:40.232270 21392 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9879e09006498eacf678c8304ead84" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33745 } } peers { permanent_uuid: "d76bcc45c09d4dd8931756897d54e4a3" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 45893 } } peers { permanent_uuid: "1cdc7121b4774eb2abaf02f91cd1ce1c" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 33187 } }
I20260618 23:28:40.232613 21398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9879e09006498eacf678c8304ead84" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33745 } } peers { permanent_uuid: "d76bcc45c09d4dd8931756897d54e4a3" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 45893 } } peers { permanent_uuid: "1cdc7121b4774eb2abaf02f91cd1ce1c" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 33187 } } }
I20260618 23:28:40.232672 21392 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:28:40.232722 21398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:28:40.233238 21320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.225471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56746 (local address 127.8.39.189:45893)
0618 23:28:40.225507 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:40.225509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:40.225518 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:28:40.225551 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:40.225551 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:40.225560 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:40.225604 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:40.226053 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.226519 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.227526 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.227723 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.227815 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:40.230345 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:40.230346 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:40.230347 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:40.230354 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:40.231780 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:40.232168 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:40.232170 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:40.232170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:40.232226 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:40.232467 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:40.232468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:40.232468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:40.232604 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:40.232769 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:40.233023 (+ 254us) server_negotiation.cc:299] Negotiation successful
0618 23:28:40.233106 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260618 23:28:40.234730 8350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260618 23:28:40.234750 21409 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 23:28:40.234807 21409 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:28:40.235059 21323 master_runner.cc:186] Error getting master registration for 127.8.39.190:33745: OK, instance_id { permanent_uuid: "6a9879e09006498eacf678c8304ead84" instance_seqno: 1781825319837932 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260618 23:28:40.237803 21323 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260618 23:28:40.245689 21396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.236380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55484 (local address 127.8.39.188:33187)
0618 23:28:40.236416 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:40.236418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:40.236595 (+ 177us) server_negotiation.cc:407] Connection header received
0618 23:28:40.236696 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:40.236697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:40.236707 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:40.236736 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:40.237539 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.238254 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.239533 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.239858 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.239942 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:40.242519 (+ 2577us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:40.242520 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:40.242521 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:40.242529 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:40.244293 (+ 1764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:40.244731 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:40.244732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:40.244733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:40.244787 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:40.244995 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:40.244996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:40.244996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:40.245195 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:40.245285 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:40.245522 (+ 237us) server_negotiation.cc:299] Negotiation successful
0618 23:28:40.245595 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260618 23:28:40.245690 21244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.236315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41172 (local address 127.8.39.190:33745)
0618 23:28:40.236356 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:40.236358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:40.236892 (+ 534us) server_negotiation.cc:407] Connection header received
0618 23:28:40.236988 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:40.236989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:40.236999 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:40.237028 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:40.238070 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.239153 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.240152 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.240455 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.240499 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:40.242519 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:40.242520 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:40.242520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:40.242529 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:40.244486 (+ 1957us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:40.244864 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:40.244865 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:40.244866 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:40.244911 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:40.245102 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:40.245102 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:40.245103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:40.245247 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:40.245312 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:40.245516 (+ 204us) server_negotiation.cc:299] Negotiation successful
0618 23:28:40.245595 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260618 23:28:40.247637 21320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.236627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56758 (local address 127.8.39.189:45893)
0618 23:28:40.236661 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:40.236663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:40.237964 (+ 1301us) server_negotiation.cc:407] Connection header received
0618 23:28:40.238034 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:40.238035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:40.238046 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:40.238093 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:40.239238 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.239986 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.241111 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.241457 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.241533 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:40.243850 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:40.243851 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:40.243852 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:40.243860 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:40.245532 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:40.246151 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:40.246152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:40.246153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:40.246206 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:40.246695 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:40.246696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:40.246696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:40.246815 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:40.247464 (+ 649us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:40.247504 (+ 40us) server_negotiation.cc:299] Negotiation successful
0618 23:28:40.247573 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260618 23:28:40.247961 21316 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.39.188" port: 33187 } attempt: 3
Jun 18 23:28:40 dist-test-slave-47qt krb5kdc[21160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825319, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.189@KRBTEST.COM for kudu/127.8.39.188@KRBTEST.COM
I20260618 23:28:40.255620 21318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.248252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:33187 (local address 127.0.0.1:55494)
0618 23:28:40.248314 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:40.248327 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:40.248379 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:40.248529 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:40.248530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:40.248684 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:28:40.248720 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:40.248721 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.249669 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.249671 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:40.250239 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:40.250241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.250631 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.250632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:40.250684 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:40.251299 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:28:40.251306 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:28:40.252891 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:28:40.254605 (+ 1714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:40.254606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:40.254606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:40.254781 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:40.254984 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:40.254984 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:40.254984 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:40.255034 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:40.255360 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:28:40.255360 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:28:40.255404 (+ 44us) client_negotiation.cc:769] Sending connection context
0618 23:28:40.255470 (+ 66us) client_negotiation.cc:240] Negotiation successful
0618 23:28:40.255554 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260618 23:28:40.255687 21396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.248334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55494 (local address 127.8.39.188:33187)
0618 23:28:40.248372 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:40.248374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:40.248382 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:28:40.248452 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:40.248453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:40.248464 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:40.248526 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:40.248803 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.249498 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.250350 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.250554 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.250635 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:40.252989 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:40.252991 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:40.252991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:40.252999 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:40.254509 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:40.254861 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:40.254862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:40.254863 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:40.254918 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:40.255087 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:40.255088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:40.255088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:40.255215 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:40.255281 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:40.255514 (+ 233us) server_negotiation.cc:299] Negotiation successful
0618 23:28:40.255593 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260618 23:28:40.257246 21316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d76bcc45c09d4dd8931756897d54e4a3: Bootstrap starting.
I20260618 23:28:40.257961 21316 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d76bcc45c09d4dd8931756897d54e4a3: Neither blocks nor log segments found. Creating new log.
I20260618 23:28:40.258278 21316 log.cc:824] T 00000000000000000000000000000000 P d76bcc45c09d4dd8931756897d54e4a3: Log is configured to *not* fsync() on all Append() calls
I20260618 23:28:40.259122 21316 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d76bcc45c09d4dd8931756897d54e4a3: No bootstrap required, opened a new log
I20260618 23:28:40.260637 21316 raft_consensus.cc:348] T 00000000000000000000000000000000 P d76bcc45c09d4dd8931756897d54e4a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9879e09006498eacf678c8304ead84" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33745 } } peers { permanent_uuid: "d76bcc45c09d4dd8931756897d54e4a3" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 45893 } } peers { permanent_uuid: "1cdc7121b4774eb2abaf02f91cd1ce1c" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 33187 } }
I20260618 23:28:40.260751 21316 raft_consensus.cc:374] T 00000000000000000000000000000000 P d76bcc45c09d4dd8931756897d54e4a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:28:40.260774 21316 raft_consensus.cc:729] T 00000000000000000000000000000000 P d76bcc45c09d4dd8931756897d54e4a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d76bcc45c09d4dd8931756897d54e4a3, State: Initialized, Role: FOLLOWER
I20260618 23:28:40.260911 21316 consensus_queue.cc:260] T 00000000000000000000000000000000 P d76bcc45c09d4dd8931756897d54e4a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9879e09006498eacf678c8304ead84" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33745 } } peers { permanent_uuid: "d76bcc45c09d4dd8931756897d54e4a3" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 45893 } } peers { permanent_uuid: "1cdc7121b4774eb2abaf02f91cd1ce1c" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 33187 } }
I20260618 23:28:40.261204 21419 sys_catalog.cc:455] T 00000000000000000000000000000000 P d76bcc45c09d4dd8931756897d54e4a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9879e09006498eacf678c8304ead84" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33745 } } peers { permanent_uuid: "d76bcc45c09d4dd8931756897d54e4a3" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 45893 } } peers { permanent_uuid: "1cdc7121b4774eb2abaf02f91cd1ce1c" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 33187 } } }
I20260618 23:28:40.261252 21316 sys_catalog.cc:564] T 00000000000000000000000000000000 P d76bcc45c09d4dd8931756897d54e4a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:28:40.261296 21419 sys_catalog.cc:458] T 00000000000000000000000000000000 P d76bcc45c09d4dd8931756897d54e4a3 [sys.catalog]: This master's current role is: FOLLOWER
W20260618 23:28:40.263103 21430 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d76bcc45c09d4dd8931756897d54e4a3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 23:28:40.263167 21430 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:28:40.263458 21248 master_runner.cc:186] Error getting master registration for 127.8.39.190:33745: OK, instance_id { permanent_uuid: "6a9879e09006498eacf678c8304ead84" instance_seqno: 1781825319837932 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260618 23:28:40.266266 21248 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260618 23:28:40.289932 21241 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.39.188" port: 33187 } attempt: 3
Jun 18 23:28:40 dist-test-slave-47qt krb5kdc[21160](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825319, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for kudu/127.8.39.188@KRBTEST.COM
I20260618 23:28:40.298763 21243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.290257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:33187 (local address 127.0.0.1:55508)
0618 23:28:40.290310 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:40.290322 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:40.290361 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:40.290573 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:40.290574 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:40.290781 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:28:40.290838 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:40.290840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.291962 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.291963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:40.292757 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:40.292759 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.293290 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.293291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:40.293354 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:40.294159 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:28:40.294167 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:28:40.295766 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:28:40.297642 (+ 1876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:40.297643 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:40.297643 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:40.297871 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:40.298096 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:40.298097 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:40.298098 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:40.298166 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:40.298438 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:28:40.298439 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:28:40.298487 (+ 48us) client_negotiation.cc:769] Sending connection context
0618 23:28:40.298560 (+ 73us) client_negotiation.cc:240] Negotiation successful
0618 23:28:40.298656 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260618 23:28:40.298768 21396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:40.290400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55508 (local address 127.8.39.188:33187)
0618 23:28:40.290449 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:40.290451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:40.290458 (+ 7us) server_negotiation.cc:407] Connection header received
0618 23:28:40.290481 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:40.290482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:40.290492 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:40.290543 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:40.290939 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.291690 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.292860 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:40.293204 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:40.293278 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:40.295857 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:40.295859 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:40.295859 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:40.295867 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:40.297553 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:40.297950 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:40.297951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:40.297952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:40.298009 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:40.298234 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:40.298235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:40.298235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:40.298367 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:40.298436 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:40.298598 (+ 162us) server_negotiation.cc:299] Negotiation successful
0618 23:28:40.298674 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260618 23:28:40.300819 21241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a9879e09006498eacf678c8304ead84: Bootstrap starting.
I20260618 23:28:40.301584 21241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6a9879e09006498eacf678c8304ead84: Neither blocks nor log segments found. Creating new log.
I20260618 23:28:40.301885 21241 log.cc:824] T 00000000000000000000000000000000 P 6a9879e09006498eacf678c8304ead84: Log is configured to *not* fsync() on all Append() calls
I20260618 23:28:40.302748 21241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6a9879e09006498eacf678c8304ead84: No bootstrap required, opened a new log
I20260618 23:28:40.304641 21241 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6a9879e09006498eacf678c8304ead84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9879e09006498eacf678c8304ead84" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33745 } } peers { permanent_uuid: "d76bcc45c09d4dd8931756897d54e4a3" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 45893 } } peers { permanent_uuid: "1cdc7121b4774eb2abaf02f91cd1ce1c" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 33187 } }
I20260618 23:28:40.304793 21241 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6a9879e09006498eacf678c8304ead84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:28:40.304829 21241 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6a9879e09006498eacf678c8304ead84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a9879e09006498eacf678c8304ead84, State: Initialized, Role: FOLLOWER
I20260618 23:28:40.304960 21241 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6a9879e09006498eacf678c8304ead84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9879e09006498eacf678c8304ead84" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33745 } } peers { permanent_uuid: "d76bcc45c09d4dd8931756897d54e4a3" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 45893 } } peers { permanent_uuid: "1cdc7121b4774eb2abaf02f91cd1ce1c" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 33187 } }
I20260618 23:28:40.305258 21433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a9879e09006498eacf678c8304ead84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9879e09006498eacf678c8304ead84" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33745 } } peers { permanent_uuid: "d76bcc45c09d4dd8931756897d54e4a3" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 45893 } } peers { permanent_uuid: "1cdc7121b4774eb2abaf02f91cd1ce1c" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 33187 } } }
I20260618 23:28:40.305384 21433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a9879e09006498eacf678c8304ead84 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:28:40.305534 21241 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6a9879e09006498eacf678c8304ead84 [sys.catalog]: configured and running, proceeding with master startup.
W20260618 23:28:40.307327 21444 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6a9879e09006498eacf678c8304ead84: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 23:28:40.307415 21444 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:28:40.310287 21173 master_runner.cc:263] No leader master found from master 6a9879e09006498eacf678c8304ead84
I20260618 23:28:40.310376 21173 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260618 23:28:40.335904 21194 tablet.cc:2369] T 00000000000000000000000000000000 P 6a9879e09006498eacf678c8304ead84: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 23:28:40.444556 21345 tablet.cc:2369] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 23:28:40.504268 21269 tablet.cc:2369] T 00000000000000000000000000000000 P d76bcc45c09d4dd8931756897d54e4a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:28:40.609503 21398 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:28:40.609637 21398 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9879e09006498eacf678c8304ead84" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33745 } } peers { permanent_uuid: "d76bcc45c09d4dd8931756897d54e4a3" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 45893 } } peers { permanent_uuid: "1cdc7121b4774eb2abaf02f91cd1ce1c" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 33187 } }
I20260618 23:28:40.610060 21398 leader_election.cc:288] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6a9879e09006498eacf678c8304ead84 (127.8.39.190:33745), d76bcc45c09d4dd8931756897d54e4a3 (127.8.39.189:45893)
I20260618 23:28:40.610471 21290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1cdc7121b4774eb2abaf02f91cd1ce1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d76bcc45c09d4dd8931756897d54e4a3" is_pre_election: true
I20260618 23:28:40.610468 21215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1cdc7121b4774eb2abaf02f91cd1ce1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a9879e09006498eacf678c8304ead84" is_pre_election: true
I20260618 23:28:40.610627 21290 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d76bcc45c09d4dd8931756897d54e4a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cdc7121b4774eb2abaf02f91cd1ce1c in term 0.
I20260618 23:28:40.610627 21215 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6a9879e09006498eacf678c8304ead84 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cdc7121b4774eb2abaf02f91cd1ce1c in term 0.
I20260618 23:28:40.610809 21341 leader_election.cc:302] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cdc7121b4774eb2abaf02f91cd1ce1c, 6a9879e09006498eacf678c8304ead84; no voters:
I20260618 23:28:40.610965 21398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:28:40.611023 21398 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:28:40.611048 21398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:28:40.612222 21398 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9879e09006498eacf678c8304ead84" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33745 } } peers { permanent_uuid: "d76bcc45c09d4dd8931756897d54e4a3" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 45893 } } peers { permanent_uuid: "1cdc7121b4774eb2abaf02f91cd1ce1c" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 33187 } }
I20260618 23:28:40.612376 21398 leader_election.cc:288] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c [CANDIDATE]: Term 1 election: Requested vote from peers 6a9879e09006498eacf678c8304ead84 (127.8.39.190:33745), d76bcc45c09d4dd8931756897d54e4a3 (127.8.39.189:45893)
I20260618 23:28:40.612598 21215 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1cdc7121b4774eb2abaf02f91cd1ce1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6a9879e09006498eacf678c8304ead84"
I20260618 23:28:40.612598 21290 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1cdc7121b4774eb2abaf02f91cd1ce1c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d76bcc45c09d4dd8931756897d54e4a3"
I20260618 23:28:40.612689 21290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d76bcc45c09d4dd8931756897d54e4a3 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:28:40.612689 21215 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6a9879e09006498eacf678c8304ead84 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:28:40.613591 21290 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d76bcc45c09d4dd8931756897d54e4a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cdc7121b4774eb2abaf02f91cd1ce1c in term 1.
I20260618 23:28:40.613603 21215 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6a9879e09006498eacf678c8304ead84 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cdc7121b4774eb2abaf02f91cd1ce1c in term 1.
I20260618 23:28:40.613811 21341 leader_election.cc:302] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cdc7121b4774eb2abaf02f91cd1ce1c, d76bcc45c09d4dd8931756897d54e4a3; no voters:
I20260618 23:28:40.613922 21398 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:28:40.614042 21398 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c [term 1 LEADER]: Becoming Leader. State: Replica: 1cdc7121b4774eb2abaf02f91cd1ce1c, State: Running, Role: LEADER
I20260618 23:28:40.614137 21398 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9879e09006498eacf678c8304ead84" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33745 } } peers { permanent_uuid: "d76bcc45c09d4dd8931756897d54e4a3" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 45893 } } peers { permanent_uuid: "1cdc7121b4774eb2abaf02f91cd1ce1c" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 33187 } }
I20260618 23:28:40.614565 21446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cdc7121b4774eb2abaf02f91cd1ce1c. Latest consensus state: current_term: 1 leader_uuid: "1cdc7121b4774eb2abaf02f91cd1ce1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9879e09006498eacf678c8304ead84" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33745 } } peers { permanent_uuid: "d76bcc45c09d4dd8931756897d54e4a3" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 45893 } } peers { permanent_uuid: "1cdc7121b4774eb2abaf02f91cd1ce1c" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 33187 } } }
I20260618 23:28:40.614655 21446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c [sys.catalog]: This master's current role is: LEADER
I20260618 23:28:40.614859 21448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:28:40.615042 21448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:28:40.616449 21290 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d76bcc45c09d4dd8931756897d54e4a3 [term 1 FOLLOWER]: Refusing update from remote peer 1cdc7121b4774eb2abaf02f91cd1ce1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:28:40.616729 21215 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6a9879e09006498eacf678c8304ead84 [term 1 FOLLOWER]: Refusing update from remote peer 1cdc7121b4774eb2abaf02f91cd1ce1c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:28:40.616822 21398 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d76bcc45c09d4dd8931756897d54e4a3" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 45893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:28:40.617218 21398 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6a9879e09006498eacf678c8304ead84" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33745 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:28:40.618440 21419 sys_catalog.cc:455] T 00000000000000000000000000000000 P d76bcc45c09d4dd8931756897d54e4a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cdc7121b4774eb2abaf02f91cd1ce1c. Latest consensus state: current_term: 1 leader_uuid: "1cdc7121b4774eb2abaf02f91cd1ce1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9879e09006498eacf678c8304ead84" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33745 } } peers { permanent_uuid: "d76bcc45c09d4dd8931756897d54e4a3" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 45893 } } peers { permanent_uuid: "1cdc7121b4774eb2abaf02f91cd1ce1c" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 33187 } } }
I20260618 23:28:40.618537 21419 sys_catalog.cc:458] T 00000000000000000000000000000000 P d76bcc45c09d4dd8931756897d54e4a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:28:40.618602 21433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a9879e09006498eacf678c8304ead84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cdc7121b4774eb2abaf02f91cd1ce1c. Latest consensus state: current_term: 1 leader_uuid: "1cdc7121b4774eb2abaf02f91cd1ce1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9879e09006498eacf678c8304ead84" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33745 } } peers { permanent_uuid: "d76bcc45c09d4dd8931756897d54e4a3" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 45893 } } peers { permanent_uuid: "1cdc7121b4774eb2abaf02f91cd1ce1c" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 33187 } } }
I20260618 23:28:40.618698 21433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a9879e09006498eacf678c8304ead84 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:28:40.619431 21446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1cdc7121b4774eb2abaf02f91cd1ce1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9879e09006498eacf678c8304ead84" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33745 } } peers { permanent_uuid: "d76bcc45c09d4dd8931756897d54e4a3" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 45893 } } peers { permanent_uuid: "1cdc7121b4774eb2abaf02f91cd1ce1c" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 33187 } } }
I20260618 23:28:40.619467 21398 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1cdc7121b4774eb2abaf02f91cd1ce1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9879e09006498eacf678c8304ead84" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33745 } } peers { permanent_uuid: "d76bcc45c09d4dd8931756897d54e4a3" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 45893 } } peers { permanent_uuid: "1cdc7121b4774eb2abaf02f91cd1ce1c" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 33187 } } }
I20260618 23:28:40.619570 21398 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c [sys.catalog]: This master's current role is: LEADER
I20260618 23:28:40.619597 21450 mvcc.cc:205] Tried to move back new op lower bound from 7298356513241739264 to 7298356513235439616. Current Snapshot: MvccSnapshot[applied={T|T < 7298356513241739264}]
I20260618 23:28:40.619575 21449 mvcc.cc:205] Tried to move back new op lower bound from 7298356513241739264 to 7298356513235439616. Current Snapshot: MvccSnapshot[applied={T|T < 7298356513241739264}]
I20260618 23:28:40.619531 21446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c [sys.catalog]: This master's current role is: LEADER
I20260618 23:28:40.619632 21433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6a9879e09006498eacf678c8304ead84 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1cdc7121b4774eb2abaf02f91cd1ce1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9879e09006498eacf678c8304ead84" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33745 } } peers { permanent_uuid: "d76bcc45c09d4dd8931756897d54e4a3" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 45893 } } peers { permanent_uuid: "1cdc7121b4774eb2abaf02f91cd1ce1c" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 33187 } } }
I20260618 23:28:40.619700 21433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6a9879e09006498eacf678c8304ead84 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:28:40.619681 21419 sys_catalog.cc:455] T 00000000000000000000000000000000 P d76bcc45c09d4dd8931756897d54e4a3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1cdc7121b4774eb2abaf02f91cd1ce1c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a9879e09006498eacf678c8304ead84" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33745 } } peers { permanent_uuid: "d76bcc45c09d4dd8931756897d54e4a3" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 45893 } } peers { permanent_uuid: "1cdc7121b4774eb2abaf02f91cd1ce1c" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 33187 } } }
I20260618 23:28:40.619750 21419 sys_catalog.cc:458] T 00000000000000000000000000000000 P d76bcc45c09d4dd8931756897d54e4a3 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:28:40.619719 21451 mvcc.cc:205] Tried to move back new op lower bound from 7298356513241739264 to 7298356513235439616. Current Snapshot: MvccSnapshot[applied={T|T < 7298356513241739264}]
I20260618 23:28:40.620822 21448 catalog_manager.cc:1347] Generated new cluster ID: cf008a7ee28a41b3888911a667461851
I20260618 23:28:40.620869 21448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:28:40.626807 21448 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:28:40.627666 21448 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:28:40.636504 21448 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1cdc7121b4774eb2abaf02f91cd1ce1c: Generated new TSK 0
I20260618 23:28:40.636729 21448 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:28:41.263613 21430 catalog_manager.cc:1259] Loaded cluster ID: cf008a7ee28a41b3888911a667461851
I20260618 23:28:41.263695 21430 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d76bcc45c09d4dd8931756897d54e4a3: loading cluster ID for follower catalog manager: success
I20260618 23:28:41.264588 21430 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d76bcc45c09d4dd8931756897d54e4a3: acquiring CA information for follower catalog manager: success
I20260618 23:28:41.264904 21430 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d76bcc45c09d4dd8931756897d54e4a3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260618 23:28:41.307879 21444 catalog_manager.cc:1259] Loaded cluster ID: cf008a7ee28a41b3888911a667461851
I20260618 23:28:41.307950 21444 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6a9879e09006498eacf678c8304ead84: loading cluster ID for follower catalog manager: success
I20260618 23:28:41.308781 21444 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6a9879e09006498eacf678c8304ead84: acquiring CA information for follower catalog manager: success
I20260618 23:28:41.309082 21444 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6a9879e09006498eacf678c8304ead84: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260618 23:28:41.711346 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 21173
I20260618 23:28:41.716629 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 21248
W20260618 23:28:41.721597 21341 connection.cc:535] client connection to 127.8.39.189:45893 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260618 23:28:41.721736 21341 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260618 23:28:41.722074 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 21323
2026-06-18T23:28:41Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3197 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:28:41 dist-test-slave-47qt krb5kdc[21465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:28:41 dist-test-slave-47qt krb5kdc[21465](info): set up 2 sockets
Jun 18 23:28:41 dist-test-slave-47qt krb5kdc[21465](info): commencing operation
krb5kdc: starting...
W20260618 23:28:42.841238 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:28:42 dist-test-slave-47qt krb5kdc[21465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:28:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:28:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.39.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
I20260618 23:28:42.973075 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:43345
--webserver_interface=127.8.39.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:45529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:43345
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:28:43.070215 21478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:28:43.070436 21478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:28:43.070464 21478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:28:43.071606 21478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:28:43.071642 21478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:28:43.071657 21478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:28:43.071669 21478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:28:43.071681 21478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:28:43.073181 21478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:45529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:43345
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:43345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:28:43.073385 21478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:28:43.073563 21478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:28:43.079043 21484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:43.078960 21486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:43.079073 21483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:28:43.079236 21478 server_base.cc:1034] running on GCE node
I20260618 23:28:43.079612 21478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:28:43.080056 21478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:28:43.081230 21478 hybrid_clock.cc:648] HybridClock initialized: now 1781825323081194 us; error 70 us; skew 500 ppm
Jun 18 23:28:43 dist-test-slave-47qt krb5kdc[21465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825323, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:28:43.083860 21478 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:28:43.084318 21478 webserver.cc:460] Webserver started at http://127.8.39.190:35431/ using document root <none> and password file <none>
I20260618 23:28:43.084574 21478 fs_manager.cc:359] Metadata directory not provided
I20260618 23:28:43.084645 21478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:28:43.084789 21478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:28:43.085901 21478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "a2f63e78e00244b8b4488365fb955910"
format_stamp: "Formatted at 2026-06-18 23:28:43 on dist-test-slave-47qt"
server_key: "681549528b24ce0760198a243d8a7087"
server_key_iv: "d28f24148bae63e549cdc7f2bc27ea5b"
server_key_version: "encryptionkey@0"
I20260618 23:28:43.086261 21478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "a2f63e78e00244b8b4488365fb955910"
format_stamp: "Formatted at 2026-06-18 23:28:43 on dist-test-slave-47qt"
server_key: "681549528b24ce0760198a243d8a7087"
server_key_iv: "d28f24148bae63e549cdc7f2bc27ea5b"
server_key_version: "encryptionkey@0"
I20260618 23:28:43.087589 21478 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260618 23:28:43.088586 21493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:43.088852 21478 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:28:43.088958 21478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "a2f63e78e00244b8b4488365fb955910"
format_stamp: "Formatted at 2026-06-18 23:28:43 on dist-test-slave-47qt"
server_key: "681549528b24ce0760198a243d8a7087"
server_key_iv: "d28f24148bae63e549cdc7f2bc27ea5b"
server_key_version: "encryptionkey@0"
I20260618 23:28:43.089051 21478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:28:43.142376 21478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:28:43.144456 21478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:28:43.144629 21478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:28:43.148726 21478 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:43345
I20260618 23:28:43.148761 21545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:43345 every 8 connection(s)
I20260618 23:28:43.149479 21478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:28:43.150559 21546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:28:43.152840 21546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2f63e78e00244b8b4488365fb955910: Bootstrap starting.
I20260618 23:28:43.153467 21546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2f63e78e00244b8b4488365fb955910: Neither blocks nor log segments found. Creating new log.
I20260618 23:28:43.153769 21546 log.cc:824] T 00000000000000000000000000000000 P a2f63e78e00244b8b4488365fb955910: Log is configured to *not* fsync() on all Append() calls
I20260618 23:28:43.154525 21546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2f63e78e00244b8b4488365fb955910: No bootstrap required, opened a new log
I20260618 23:28:43.156162 21546 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2f63e78e00244b8b4488365fb955910 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f63e78e00244b8b4488365fb955910" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 43345 } }
I20260618 23:28:43.156303 21546 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2f63e78e00244b8b4488365fb955910 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:28:43.156327 21546 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2f63e78e00244b8b4488365fb955910 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f63e78e00244b8b4488365fb955910, State: Initialized, Role: FOLLOWER
I20260618 23:28:43.156476 21546 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2f63e78e00244b8b4488365fb955910 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f63e78e00244b8b4488365fb955910" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 43345 } }
I20260618 23:28:43.156553 21546 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2f63e78e00244b8b4488365fb955910 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:28:43.156574 21546 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2f63e78e00244b8b4488365fb955910 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:28:43.156641 21546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2f63e78e00244b8b4488365fb955910 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:28:43.157269 21546 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2f63e78e00244b8b4488365fb955910 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f63e78e00244b8b4488365fb955910" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 43345 } }
I20260618 23:28:43.157356 21546 leader_election.cc:302] T 00000000000000000000000000000000 P a2f63e78e00244b8b4488365fb955910 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2f63e78e00244b8b4488365fb955910; no voters:
I20260618 23:28:43.157528 21546 leader_election.cc:288] T 00000000000000000000000000000000 P a2f63e78e00244b8b4488365fb955910 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:28:43.157601 21549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2f63e78e00244b8b4488365fb955910 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:28:43.157823 21546 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2f63e78e00244b8b4488365fb955910 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:28:43.157826 21549 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2f63e78e00244b8b4488365fb955910 [term 1 LEADER]: Becoming Leader. State: Replica: a2f63e78e00244b8b4488365fb955910, State: Running, Role: LEADER
I20260618 23:28:43.158010 21549 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2f63e78e00244b8b4488365fb955910 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f63e78e00244b8b4488365fb955910" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 43345 } }
I20260618 23:28:43.158447 21550 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2f63e78e00244b8b4488365fb955910 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2f63e78e00244b8b4488365fb955910" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f63e78e00244b8b4488365fb955910" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 43345 } } }
I20260618 23:28:43.158527 21550 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2f63e78e00244b8b4488365fb955910 [sys.catalog]: This master's current role is: LEADER
I20260618 23:28:43.158595 21551 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2f63e78e00244b8b4488365fb955910 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2f63e78e00244b8b4488365fb955910. Latest consensus state: current_term: 1 leader_uuid: "a2f63e78e00244b8b4488365fb955910" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f63e78e00244b8b4488365fb955910" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 43345 } } }
I20260618 23:28:43.158676 21551 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2f63e78e00244b8b4488365fb955910 [sys.catalog]: This master's current role is: LEADER
I20260618 23:28:43.159202 21558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:28:43.159616 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 21478
I20260618 23:28:43.159724 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:28:43.159777 21558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:28:43.160145 8350 external_mini_cluster.cc:1428] Setting key 423f6378a10ee42d4a33a00e17a05aad
I20260618 23:28:43.163727 21558 catalog_manager.cc:1347] Generated new cluster ID: 5d59bc2bbc8448ea9cea1f0a466fafc2
I20260618 23:28:43.163810 21558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 18 23:28:43 dist-test-slave-47qt krb5kdc[21465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825322, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:28:43.172633 21568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:43.160744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55492 (local address 127.8.39.190:43345)
0618 23:28:43.161258 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:43.161269 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:43.161309 (+ 40us) server_negotiation.cc:407] Connection header received
0618 23:28:43.161447 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:43.161455 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:43.161516 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:43.161871 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:43.163197 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:43.164360 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:43.165206 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:43.165654 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:43.165780 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:43.168751 (+ 2971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:43.168757 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:43.168758 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:43.168775 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:43.170977 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:43.171414 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:43.171417 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:43.171418 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:43.171523 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:43.171751 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:43.171752 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:43.171753 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:43.171998 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:43.172199 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:43.172270 (+ 71us) server_negotiation.cc:299] Negotiation successful
0618 23:28:43.172413 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":115,"threads_started":1}
I20260618 23:28:43.173166 21558 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:28:43.173740 8350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:28:43.173959 21558 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:28:43.186574 21558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2f63e78e00244b8b4488365fb955910: Generated new TSK 0
I20260618 23:28:43.186802 21558 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:28:43.228242 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 21478
2026-06-18T23:28:43Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1502 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:28:43 dist-test-slave-47qt krb5kdc[21573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:28:43 dist-test-slave-47qt krb5kdc[21573](info): set up 2 sockets
Jun 18 23:28:43 dist-test-slave-47qt krb5kdc[21573](info): commencing operation
krb5kdc: starting...
W20260618 23:28:44.301245 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:28:44 dist-test-slave-47qt krb5kdc[21573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:28:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:28:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.39.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
I20260618 23:28:44.420177 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:41745
--webserver_interface=127.8.39.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:41745
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:28:44.505059 21586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:28:44.505307 21586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:28:44.505331 21586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:28:44.506502 21586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:28:44.506556 21586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:28:44.506572 21586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:28:44.506585 21586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:28:44.506597 21586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:28:44.508157 21586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:41745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:41745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:28:44.508422 21586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:28:44.508666 21586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:28:44.513773 21591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:44.513780 21594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:28:44.513854 21586 server_base.cc:1034] running on GCE node
W20260618 23:28:44.513779 21592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:28:44.514269 21586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:28:44.514576 21586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:28:44.515749 21586 hybrid_clock.cc:648] HybridClock initialized: now 1781825324515748 us; error 39 us; skew 500 ppm
Jun 18 23:28:44 dist-test-slave-47qt krb5kdc[21573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825324, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:28:44.518322 21586 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:28:44.518713 21586 webserver.cc:460] Webserver started at http://127.8.39.190:46293/ using document root <none> and password file <none>
I20260618 23:28:44.518886 21586 fs_manager.cc:359] Metadata directory not provided
I20260618 23:28:44.518925 21586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:28:44.519011 21586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:28:44.519981 21586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "b0ea7459b1874d59b31a19576c346dca"
format_stamp: "Formatted at 2026-06-18 23:28:44 on dist-test-slave-47qt"
server_key: "437c6fcb71deec0cbafc9324a92fcb6f"
server_key_iv: "30288963de06b5eae6b057536d2e63f1"
server_key_version: "encryptionkey@0"
I20260618 23:28:44.520277 21586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "b0ea7459b1874d59b31a19576c346dca"
format_stamp: "Formatted at 2026-06-18 23:28:44 on dist-test-slave-47qt"
server_key: "437c6fcb71deec0cbafc9324a92fcb6f"
server_key_iv: "30288963de06b5eae6b057536d2e63f1"
server_key_version: "encryptionkey@0"
I20260618 23:28:44.521596 21586 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:28:44.522325 21601 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:44.522832 21586 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:28:44.522918 21586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "b0ea7459b1874d59b31a19576c346dca"
format_stamp: "Formatted at 2026-06-18 23:28:44 on dist-test-slave-47qt"
server_key: "437c6fcb71deec0cbafc9324a92fcb6f"
server_key_iv: "30288963de06b5eae6b057536d2e63f1"
server_key_version: "encryptionkey@0"
I20260618 23:28:44.522977 21586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:28:44.550873 21586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:28:44.552973 21586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:28:44.553095 21586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:28:44.557415 21586 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:41745
I20260618 23:28:44.557497 21653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:41745 every 8 connection(s)
I20260618 23:28:44.557874 21586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:28:44.558955 21654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:28:44.561334 21654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca: Bootstrap starting.
I20260618 23:28:44.561928 21654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca: Neither blocks nor log segments found. Creating new log.
I20260618 23:28:44.562215 21654 log.cc:824] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca: Log is configured to *not* fsync() on all Append() calls
I20260618 23:28:44.562912 21654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca: No bootstrap required, opened a new log
I20260618 23:28:44.564488 21654 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ea7459b1874d59b31a19576c346dca" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41745 } }
I20260618 23:28:44.564625 21654 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:28:44.564648 21654 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ea7459b1874d59b31a19576c346dca, State: Initialized, Role: FOLLOWER
I20260618 23:28:44.564837 21654 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ea7459b1874d59b31a19576c346dca" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41745 } }
I20260618 23:28:44.564920 21654 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:28:44.564986 21654 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:28:44.565048 21654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:28:44.565485 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 21586
I20260618 23:28:44.565579 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:28:44.565761 8350 external_mini_cluster.cc:1428] Setting key 695645e15bf4c62690d6b90e8305e145
I20260618 23:28:44.566207 21654 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ea7459b1874d59b31a19576c346dca" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41745 } }
I20260618 23:28:44.566331 21654 leader_election.cc:302] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0ea7459b1874d59b31a19576c346dca; no voters:
I20260618 23:28:44.566762 21658 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:28:44.566817 21654 leader_election.cc:288] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:28:44.566994 21658 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [term 1 LEADER]: Becoming Leader. State: Replica: b0ea7459b1874d59b31a19576c346dca, State: Running, Role: LEADER
I20260618 23:28:44.567148 21658 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ea7459b1874d59b31a19576c346dca" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41745 } }
I20260618 23:28:44.567255 21654 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:28:44.567554 21660 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0ea7459b1874d59b31a19576c346dca. Latest consensus state: current_term: 1 leader_uuid: "b0ea7459b1874d59b31a19576c346dca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ea7459b1874d59b31a19576c346dca" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41745 } } }
I20260618 23:28:44.567651 21660 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [sys.catalog]: This master's current role is: LEADER
I20260618 23:28:44.567549 21658 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0ea7459b1874d59b31a19576c346dca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ea7459b1874d59b31a19576c346dca" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41745 } } }
I20260618 23:28:44.567931 21658 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [sys.catalog]: This master's current role is: LEADER
I20260618 23:28:44.568353 21664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:28:44.568907 21664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:28:44.570729 21664 catalog_manager.cc:1347] Generated new cluster ID: 4c8069eb6e0a42d4a48c413ff14e0888
I20260618 23:28:44.570783 21664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 18 23:28:44 dist-test-slave-47qt krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:28:44.579883 21659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:44.566701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56574 (local address 127.8.39.190:41745)
0618 23:28:44.567034 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:44.567045 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:44.567095 (+ 50us) server_negotiation.cc:407] Connection header received
0618 23:28:44.567229 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:44.567237 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:44.567295 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:44.568290 (+ 995us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:44.568407 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:44.569908 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:44.570955 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:44.571574 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:44.571690 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:44.575757 (+ 4067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:44.575763 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:44.575764 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:44.575778 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:44.578148 (+ 2370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:44.578615 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:44.578618 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:44.578618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:44.578731 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:44.578955 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:44.578956 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:44.578957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:44.579255 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:44.579399 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:44.579538 (+ 139us) server_negotiation.cc:299] Negotiation successful
0618 23:28:44.579666 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":108,"threads_started":1}
I20260618 23:28:44.583148 21664 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:28:44.583642 21664 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:28:44.586112 21664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca: Generated new TSK 0
I20260618 23:28:44.586313 21664 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.39.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
I20260618 23:28:44.615882 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.129:0
--local_ip_for_outbound_sockets=127.8.39.129
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:41745
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:28:44.700704 21680 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:28:44.700903 21680 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:28:44.700928 21680 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:28:44.702138 21680 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:28:44.702180 21680 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:28:44.702208 21680 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.129
I20260618 23:28:44.703670 21680 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:41745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21680
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.129
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:28:44.703922 21680 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:28:44.704173 21680 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:28:44.709415 21685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:44.709443 21686 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:44.709419 21688 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:28:44.709676 21680 server_base.cc:1034] running on GCE node
I20260618 23:28:44.709882 21680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:28:44.710145 21680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:28:44.711331 21680 hybrid_clock.cc:648] HybridClock initialized: now 1781825324711331 us; error 48 us; skew 500 ppm
Jun 18 23:28:44 dist-test-slave-47qt krb5kdc[21573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825324, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:28:44.713996 21680 init.cc:377] Logged in from keytab as kudu/127.8.39.129@KRBTEST.COM (short username kudu)
I20260618 23:28:44.714381 21680 webserver.cc:460] Webserver started at http://127.8.39.129:33481/ using document root <none> and password file <none>
I20260618 23:28:44.714586 21680 fs_manager.cc:359] Metadata directory not provided
I20260618 23:28:44.714628 21680 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:28:44.714717 21680 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:28:44.715716 21680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/data/instance:
uuid: "53e617757a0b49e4961839bae4389d7c"
format_stamp: "Formatted at 2026-06-18 23:28:44 on dist-test-slave-47qt"
server_key: "b1003fcb2581ac6d36e977d9574190b9"
server_key_iv: "b7f9506bc75ebb0a1ac891376cbdda1b"
server_key_version: "encryptionkey@0"
I20260618 23:28:44.716012 21680 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance:
uuid: "53e617757a0b49e4961839bae4389d7c"
format_stamp: "Formatted at 2026-06-18 23:28:44 on dist-test-slave-47qt"
server_key: "b1003fcb2581ac6d36e977d9574190b9"
server_key_iv: "b7f9506bc75ebb0a1ac891376cbdda1b"
server_key_version: "encryptionkey@0"
I20260618 23:28:44.717331 21680 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260618 23:28:44.718052 21695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:44.718289 21680 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:28:44.718374 21680 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/wal
uuid: "53e617757a0b49e4961839bae4389d7c"
format_stamp: "Formatted at 2026-06-18 23:28:44 on dist-test-slave-47qt"
server_key: "b1003fcb2581ac6d36e977d9574190b9"
server_key_iv: "b7f9506bc75ebb0a1ac891376cbdda1b"
server_key_version: "encryptionkey@0"
I20260618 23:28:44.718443 21680 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:28:44.738157 21680 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:28:44.740139 21680 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:28:44.740257 21680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:28:44.740468 21680 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:28:44.740819 21680 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:28:44.740870 21680 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:44.740923 21680 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:28:44.740950 21680 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:44.747284 21680 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.129:45297
I20260618 23:28:44.747373 21808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.129:45297 every 8 connection(s)
I20260618 23:28:44.747699 21680 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
I20260618 23:28:44.751098 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 21680
I20260618 23:28:44.751189 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance
I20260618 23:28:44.751402 8350 external_mini_cluster.cc:1428] Setting key 9b2a15e10fab86471cc35df37d6bba93
Jun 18 23:28:44 dist-test-slave-47qt krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825324, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:28:44.757983 21659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:44.748512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:35969 (local address 127.8.39.190:41745)
0618 23:28:44.748658 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:44.748660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:44.748940 (+ 280us) server_negotiation.cc:407] Connection header received
0618 23:28:44.749551 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:44.749553 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:44.749573 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:44.749679 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:44.750731 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:44.751335 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:44.752591 (+ 1256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:44.752812 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:44.752887 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:44.755076 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:44.755078 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:44.755079 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:44.755089 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:44.756518 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:44.756949 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:44.756950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:44.756951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:44.757010 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:44.757318 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:44.757319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:44.757319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:44.757437 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:44.757520 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:44.757793 (+ 273us) server_negotiation.cc:299] Negotiation successful
0618 23:28:44.757881 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260618 23:28:44.758162 21811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:44.748465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.129:35969)
0618 23:28:44.748679 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:44.748723 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:44.749387 (+ 664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:44.749778 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:44.749782 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:44.750371 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:28:44.750595 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:44.750599 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:44.751436 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:44.751437 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:44.752460 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:44.752463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:44.752907 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:44.752908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:44.753029 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:44.753626 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:28:44.753634 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:28:44.754974 (+ 1340us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:28:44.756629 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:44.756636 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:44.756638 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:44.756862 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:44.757084 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:44.757084 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:44.757085 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:44.757248 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:44.757512 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:28:44.757518 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:28:44.757658 (+ 140us) client_negotiation.cc:769] Sending connection context
0618 23:28:44.757758 (+ 100us) client_negotiation.cc:240] Negotiation successful
0618 23:28:44.757925 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":95,"threads_started":1}
I20260618 23:28:44.758877 21809 heartbeater.cc:347] Connected to a master server at 127.8.39.190:41745
I20260618 23:28:44.758960 21809 heartbeater.cc:464] Registering TS with master...
I20260618 23:28:44.759153 21809 heartbeater.cc:511] Master 127.8.39.190:41745 requested a full tablet report, sending...
I20260618 23:28:44.759776 21618 ts_manager.cc:194] Registered new tserver with Master: 53e617757a0b49e4961839bae4389d7c (127.8.39.129:45297)
WARNING: no policy specified for kudu/127.8.39.130@KRBTEST.COM; defaulting to no policy
I20260618 23:28:44.760515 21618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.129@KRBTEST.COM'} at 127.8.39.129:35969
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
I20260618 23:28:44.779718 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.130:0
--local_ip_for_outbound_sockets=127.8.39.130
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:41745
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:28:44.868618 21814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:28:44.868820 21814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:28:44.868845 21814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:28:44.870072 21814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:28:44.870116 21814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:28:44.870144 21814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.130
I20260618 23:28:44.871502 21814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:41745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.130
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:28:44.871701 21814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:28:44.871870 21814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:28:44.876659 21820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:28:44.876721 21814 server_base.cc:1034] running on GCE node
W20260618 23:28:44.876754 21819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:44.876888 21822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:28:44.877189 21814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:28:44.877421 21814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:28:44.878594 21814 hybrid_clock.cc:648] HybridClock initialized: now 1781825324878577 us; error 28 us; skew 500 ppm
Jun 18 23:28:44 dist-test-slave-47qt krb5kdc[21573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825324, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:28:44.881078 21814 init.cc:377] Logged in from keytab as kudu/127.8.39.130@KRBTEST.COM (short username kudu)
I20260618 23:28:44.881528 21814 webserver.cc:460] Webserver started at http://127.8.39.130:43549/ using document root <none> and password file <none>
I20260618 23:28:44.881769 21814 fs_manager.cc:359] Metadata directory not provided
I20260618 23:28:44.881837 21814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:28:44.881984 21814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:28:44.883044 21814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/data/instance:
uuid: "fbd16d57fbff4156b4549511ed9319ca"
format_stamp: "Formatted at 2026-06-18 23:28:44 on dist-test-slave-47qt"
server_key: "3fe41cd30ff79803d61fc86ca00f8291"
server_key_iv: "15032fbbeac4574ce0161e3b2f178cd7"
server_key_version: "encryptionkey@0"
I20260618 23:28:44.883428 21814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance:
uuid: "fbd16d57fbff4156b4549511ed9319ca"
format_stamp: "Formatted at 2026-06-18 23:28:44 on dist-test-slave-47qt"
server_key: "3fe41cd30ff79803d61fc86ca00f8291"
server_key_iv: "15032fbbeac4574ce0161e3b2f178cd7"
server_key_version: "encryptionkey@0"
I20260618 23:28:44.884891 21814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:28:44.885783 21829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:44.885990 21814 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:28:44.886102 21814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/wal
uuid: "fbd16d57fbff4156b4549511ed9319ca"
format_stamp: "Formatted at 2026-06-18 23:28:44 on dist-test-slave-47qt"
server_key: "3fe41cd30ff79803d61fc86ca00f8291"
server_key_iv: "15032fbbeac4574ce0161e3b2f178cd7"
server_key_version: "encryptionkey@0"
I20260618 23:28:44.886180 21814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:28:44.898358 21814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:28:44.900693 21814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:28:44.900861 21814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:28:44.901090 21814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:28:44.901450 21814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:28:44.901505 21814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:44.901546 21814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:28:44.901575 21814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:44.907845 21814 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.130:46529
I20260618 23:28:44.907904 21942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.130:46529 every 8 connection(s)
I20260618 23:28:44.908249 21814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
Jun 18 23:28:44 dist-test-slave-47qt krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825324, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:28:44.915194 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 21814
I20260618 23:28:44.915292 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance
I20260618 23:28:44.915500 8350 external_mini_cluster.cc:1428] Setting key 15ce36f925ddb229fc35e2468a25a8bb
I20260618 23:28:44.918370 21659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:44.909107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:38029 (local address 127.8.39.190:41745)
0618 23:28:44.909205 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:44.909207 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:44.909397 (+ 190us) server_negotiation.cc:407] Connection header received
0618 23:28:44.909889 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:44.909890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:44.909900 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:44.909958 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:44.911009 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:44.911508 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:44.912729 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:44.912924 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:44.912985 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:44.915414 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:44.915416 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:44.915416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:44.915423 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:44.916881 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:44.917407 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:44.917408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:44.917408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:44.917467 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:44.917727 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:44.917728 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:44.917728 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:44.917842 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:44.917919 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:44.918195 (+ 276us) server_negotiation.cc:299] Negotiation successful
0618 23:28:44.918281 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260618 23:28:44.918622 21945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:44.909060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.130:38029)
0618 23:28:44.909275 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:44.909311 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:44.909784 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:44.910029 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:44.910032 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:44.910695 (+ 663us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:28:44.910884 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:44.910889 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:44.911638 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:44.911639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:44.912427 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:44.912429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:44.913017 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:44.913017 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:44.913188 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:44.913705 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:28:44.913714 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:28:44.915309 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:28:44.917010 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:44.917017 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:44.917019 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:44.917321 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:44.917534 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:44.917535 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:44.917535 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:44.917653 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:44.917910 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:28:44.917916 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:28:44.918059 (+ 143us) client_negotiation.cc:769] Sending connection context
0618 23:28:44.918162 (+ 103us) client_negotiation.cc:240] Negotiation successful
0618 23:28:44.918324 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":126,"threads_started":1}
I20260618 23:28:44.919060 21943 heartbeater.cc:347] Connected to a master server at 127.8.39.190:41745
I20260618 23:28:44.919152 21943 heartbeater.cc:464] Registering TS with master...
I20260618 23:28:44.919334 21943 heartbeater.cc:511] Master 127.8.39.190:41745 requested a full tablet report, sending...
I20260618 23:28:44.919641 21618 ts_manager.cc:194] Registered new tserver with Master: fbd16d57fbff4156b4549511ed9319ca (127.8.39.130:46529)
I20260618 23:28:44.920022 21618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.130@KRBTEST.COM'} at 127.8.39.130:38029
WARNING: no policy specified for kudu/127.8.39.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
I20260618 23:28:44.940675 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.131:0
--local_ip_for_outbound_sockets=127.8.39.131
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:41745
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:28:45.028697 21948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:28:45.028934 21948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:28:45.028963 21948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:28:45.030260 21948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:28:45.030314 21948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:28:45.030346 21948 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.131
I20260618 23:28:45.033281 21948 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:41745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.131
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:28:45.033586 21948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:28:45.033839 21948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:28:45.039383 21953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:45.039337 21954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:28:45.039528 21948 server_base.cc:1034] running on GCE node
W20260618 23:28:45.039412 21956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:28:45.039862 21948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:28:45.040124 21948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:28:45.041266 21948 hybrid_clock.cc:648] HybridClock initialized: now 1781825325041244 us; error 39 us; skew 500 ppm
Jun 18 23:28:45 dist-test-slave-47qt krb5kdc[21573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825325, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:28:45.044049 21948 init.cc:377] Logged in from keytab as kudu/127.8.39.131@KRBTEST.COM (short username kudu)
I20260618 23:28:45.044555 21948 webserver.cc:460] Webserver started at http://127.8.39.131:40087/ using document root <none> and password file <none>
I20260618 23:28:45.044788 21948 fs_manager.cc:359] Metadata directory not provided
I20260618 23:28:45.044864 21948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:28:45.045024 21948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:28:45.046636 21948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/data/instance:
uuid: "d5b08fbbb1864186965d51d9738877b8"
format_stamp: "Formatted at 2026-06-18 23:28:45 on dist-test-slave-47qt"
server_key: "4d16de477a96d86f015d8a1824dbe10f"
server_key_iv: "5936a638752d8e0b2db368433c376bff"
server_key_version: "encryptionkey@0"
I20260618 23:28:45.047070 21948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance:
uuid: "d5b08fbbb1864186965d51d9738877b8"
format_stamp: "Formatted at 2026-06-18 23:28:45 on dist-test-slave-47qt"
server_key: "4d16de477a96d86f015d8a1824dbe10f"
server_key_iv: "5936a638752d8e0b2db368433c376bff"
server_key_version: "encryptionkey@0"
I20260618 23:28:45.048508 21948 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:28:45.049417 21963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:45.049695 21948 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260618 23:28:45.049800 21948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/wal
uuid: "d5b08fbbb1864186965d51d9738877b8"
format_stamp: "Formatted at 2026-06-18 23:28:45 on dist-test-slave-47qt"
server_key: "4d16de477a96d86f015d8a1824dbe10f"
server_key_iv: "5936a638752d8e0b2db368433c376bff"
server_key_version: "encryptionkey@0"
I20260618 23:28:45.049888 21948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:28:45.082429 21948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:28:45.085072 21948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:28:45.085311 21948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:28:45.085654 21948 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:28:45.086082 21948 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:28:45.086134 21948 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:45.086189 21948 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:28:45.086210 21948 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:45.092120 21948 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.131:33769
I20260618 23:28:45.092157 22076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.131:33769 every 8 connection(s)
I20260618 23:28:45.092469 21948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
I20260618 23:28:45.096158 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 21948
I20260618 23:28:45.096232 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance
I20260618 23:28:45.096410 8350 external_mini_cluster.cc:1428] Setting key 673cf46d50bcf2452b77a0320ef1cb25
Jun 18 23:28:45 dist-test-slave-47qt krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825325, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:28:45.102919 21659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:45.093198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:37769 (local address 127.8.39.190:41745)
0618 23:28:45.093253 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:45.093255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:45.093457 (+ 202us) server_negotiation.cc:407] Connection header received
0618 23:28:45.093941 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:45.093942 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:45.093951 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:45.094010 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:45.094904 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.095389 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.096441 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.096719 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.096804 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:45.099073 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:45.099074 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:45.099074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:45.099081 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:45.100961 (+ 1880us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:45.101780 (+ 819us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:45.101781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:45.101782 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:45.101871 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:45.102223 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:45.102224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:45.102225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:45.102351 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:45.102421 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:45.102742 (+ 321us) server_negotiation.cc:299] Negotiation successful
0618 23:28:45.102830 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260618 23:28:45.103120 22079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:45.093185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.131:37769)
0618 23:28:45.093334 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:45.093374 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:45.093861 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:45.094061 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:45.094064 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:45.094623 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:28:45.094822 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:45.094825 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.095471 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.095471 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:45.096339 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:45.096342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.096813 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.096814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:45.096951 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:45.097504 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:28:45.097512 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:28:45.098955 (+ 1443us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:28:45.101106 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:45.101113 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:45.101147 (+ 34us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:45.101485 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:45.102006 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:45.102007 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:45.102007 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:45.102142 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:45.102424 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:28:45.102431 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:28:45.102571 (+ 140us) client_negotiation.cc:769] Sending connection context
0618 23:28:45.102706 (+ 135us) client_negotiation.cc:240] Negotiation successful
0618 23:28:45.102872 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":84,"threads_started":1}
I20260618 23:28:45.104192 22077 heartbeater.cc:347] Connected to a master server at 127.8.39.190:41745
I20260618 23:28:45.104317 22077 heartbeater.cc:464] Registering TS with master...
I20260618 23:28:45.104525 22077 heartbeater.cc:511] Master 127.8.39.190:41745 requested a full tablet report, sending...
I20260618 23:28:45.105057 21618 ts_manager.cc:194] Registered new tserver with Master: d5b08fbbb1864186965d51d9738877b8 (127.8.39.131:33769)
I20260618 23:28:45.105463 21618 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.131@KRBTEST.COM'} at 127.8.39.131:37769
I20260618 23:28:45.108132 8350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:28:45.116402 21659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:45.109624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56590 (local address 127.8.39.190:41745)
0618 23:28:45.109664 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:45.109666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:45.109764 (+ 98us) server_negotiation.cc:407] Connection header received
0618 23:28:45.109784 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:45.109785 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:45.109793 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:45.109846 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:45.110403 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.110885 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.111609 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.111800 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.111873 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:45.113148 (+ 1275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:45.113150 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:45.113150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:45.113167 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:45.114823 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:45.115380 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:45.115382 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:45.115382 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:45.115442 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:45.115684 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:45.115685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:45.115685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:45.115882 (+ 197us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:45.116002 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:45.116139 (+ 137us) server_negotiation.cc:299] Negotiation successful
0618 23:28:45.116240 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260618 23:28:45.118240 21618 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56590:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:28:45.119809 21618 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:28:45.126204 22088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:45.122155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:45297 (local address 127.0.0.1:46210)
0618 23:28:45.122470 (+ 315us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:45.122485 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:45.122551 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:45.122776 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:45.122777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:45.122778 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:28:45.122857 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:45.122859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.123867 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.123868 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:45.124757 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:45.124760 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.125869 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.125870 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:45.125960 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:45.125973 (+ 13us) client_negotiation.cc:769] Sending connection context
0618 23:28:45.126048 (+ 75us) client_negotiation.cc:240] Negotiation successful
0618 23:28:45.126107 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":7,"thread_start_us":80,"threads_started":1}
I20260618 23:28:45.126221 22091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:45.122270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46210 (local address 127.8.39.129:45297)
0618 23:28:45.122451 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:45.122456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:45.122639 (+ 183us) server_negotiation.cc:407] Connection header received
0618 23:28:45.122690 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:45.122693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:45.122709 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:45.122781 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:28:45.122940 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.123770 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.124898 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.125654 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.125740 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:45.125795 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:45.126066 (+ 271us) server_negotiation.cc:299] Negotiation successful
0618 23:28:45.126111 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":77,"threads_started":1}
I20260618 23:28:45.126323 22092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:45.122358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:33769 (local address 127.0.0.1:59204)
0618 23:28:45.122587 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:45.122596 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:45.122640 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:45.123122 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:45.123124 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:45.123126 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:28:45.123165 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:45.123167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.124115 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.124116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:45.125256 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:45.125258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.126134 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.126135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:45.126192 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:45.126196 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:28:45.126226 (+ 30us) client_negotiation.cc:240] Negotiation successful
0618 23:28:45.126264 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":65,"threads_started":1}
I20260618 23:28:45.126587 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:45.122452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59204 (local address 127.8.39.131:33769)
0618 23:28:45.122895 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:45.122899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:45.122909 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:28:45.122963 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:45.122965 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:45.122981 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:45.123055 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:28:45.123227 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.124003 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.125338 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.126034 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.126394 (+ 360us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:45.126454 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:45.126478 (+ 24us) server_negotiation.cc:299] Negotiation successful
0618 23:28:45.126511 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":410,"thread_start_us":76,"threads_started":1}
I20260618 23:28:45.127048 21743 tablet_service.cc:1467] Processing CreateTablet for tablet 2d22d1d282d24875a54526b044ab0958 (DEFAULT_TABLE table=test-table [id=8ec18b477ff04671b5909b34f2d06546]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:28:45.127110 22011 tablet_service.cc:1467] Processing CreateTablet for tablet 2d22d1d282d24875a54526b044ab0958 (DEFAULT_TABLE table=test-table [id=8ec18b477ff04671b5909b34f2d06546]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:28:45.127381 21743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d22d1d282d24875a54526b044ab0958. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:28:45.127408 22011 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d22d1d282d24875a54526b044ab0958. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:28:45.130012 22094 tablet_bootstrap.cc:492] T 2d22d1d282d24875a54526b044ab0958 P d5b08fbbb1864186965d51d9738877b8: Bootstrap starting.
I20260618 23:28:45.130028 22095 tablet_bootstrap.cc:492] T 2d22d1d282d24875a54526b044ab0958 P 53e617757a0b49e4961839bae4389d7c: Bootstrap starting.
I20260618 23:28:45.130626 22094 tablet_bootstrap.cc:654] T 2d22d1d282d24875a54526b044ab0958 P d5b08fbbb1864186965d51d9738877b8: Neither blocks nor log segments found. Creating new log.
I20260618 23:28:45.130722 22095 tablet_bootstrap.cc:654] T 2d22d1d282d24875a54526b044ab0958 P 53e617757a0b49e4961839bae4389d7c: Neither blocks nor log segments found. Creating new log.
I20260618 23:28:45.130916 22089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:45.122155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:46529 (local address 127.0.0.1:42564)
0618 23:28:45.122439 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:45.122483 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:45.122552 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:45.122772 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:45.122775 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:45.122777 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:28:45.122857 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:45.122860 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.124984 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.124985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:45.125884 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:45.125886 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.130633 (+ 4747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.130634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:45.130703 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:45.130708 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:28:45.130781 (+ 73us) client_negotiation.cc:240] Negotiation successful
0618 23:28:45.130828 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"thread_start_us":68,"threads_started":1}
I20260618 23:28:45.131225 22090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:45.122270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42564 (local address 127.8.39.130:46529)
0618 23:28:45.122527 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:45.122532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:45.122542 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:28:45.122658 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:45.122661 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:45.122677 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:45.122759 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:28:45.123868 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.124872 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.125966 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.130532 (+ 4566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.130613 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:45.130668 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:45.131085 (+ 417us) server_negotiation.cc:299] Negotiation successful
0618 23:28:45.131148 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":227,"thread_start_us":72,"threads_started":1}
I20260618 23:28:45.131220 22094 log.cc:824] T 2d22d1d282d24875a54526b044ab0958 P d5b08fbbb1864186965d51d9738877b8: Log is configured to *not* fsync() on all Append() calls
I20260618 23:28:45.131310 22095 log.cc:824] T 2d22d1d282d24875a54526b044ab0958 P 53e617757a0b49e4961839bae4389d7c: Log is configured to *not* fsync() on all Append() calls
I20260618 23:28:45.131848 21877 tablet_service.cc:1467] Processing CreateTablet for tablet 2d22d1d282d24875a54526b044ab0958 (DEFAULT_TABLE table=test-table [id=8ec18b477ff04671b5909b34f2d06546]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:28:45.131956 22095 tablet_bootstrap.cc:492] T 2d22d1d282d24875a54526b044ab0958 P 53e617757a0b49e4961839bae4389d7c: No bootstrap required, opened a new log
I20260618 23:28:45.131999 22095 ts_tablet_manager.cc:1397] T 2d22d1d282d24875a54526b044ab0958 P 53e617757a0b49e4961839bae4389d7c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:28:45.132141 21877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2d22d1d282d24875a54526b044ab0958. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:28:45.133983 22095 raft_consensus.cc:348] T 2d22d1d282d24875a54526b044ab0958 P 53e617757a0b49e4961839bae4389d7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e617757a0b49e4961839bae4389d7c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45297 } } peers { permanent_uuid: "fbd16d57fbff4156b4549511ed9319ca" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46529 } } peers { permanent_uuid: "d5b08fbbb1864186965d51d9738877b8" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 33769 } }
I20260618 23:28:45.134153 22095 raft_consensus.cc:374] T 2d22d1d282d24875a54526b044ab0958 P 53e617757a0b49e4961839bae4389d7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:28:45.134207 22095 raft_consensus.cc:729] T 2d22d1d282d24875a54526b044ab0958 P 53e617757a0b49e4961839bae4389d7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e617757a0b49e4961839bae4389d7c, State: Initialized, Role: FOLLOWER
I20260618 23:28:45.134321 22095 consensus_queue.cc:260] T 2d22d1d282d24875a54526b044ab0958 P 53e617757a0b49e4961839bae4389d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e617757a0b49e4961839bae4389d7c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45297 } } peers { permanent_uuid: "fbd16d57fbff4156b4549511ed9319ca" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46529 } } peers { permanent_uuid: "d5b08fbbb1864186965d51d9738877b8" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 33769 } }
I20260618 23:28:45.134580 22095 ts_tablet_manager.cc:1428] T 2d22d1d282d24875a54526b044ab0958 P 53e617757a0b49e4961839bae4389d7c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:28:45.134719 21809 heartbeater.cc:503] Master 127.8.39.190:41745 was elected leader, sending a full tablet report...
I20260618 23:28:45.135633 22094 tablet_bootstrap.cc:492] T 2d22d1d282d24875a54526b044ab0958 P d5b08fbbb1864186965d51d9738877b8: No bootstrap required, opened a new log
I20260618 23:28:45.135612 22099 tablet_bootstrap.cc:492] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca: Bootstrap starting.
I20260618 23:28:45.135728 22094 ts_tablet_manager.cc:1397] T 2d22d1d282d24875a54526b044ab0958 P d5b08fbbb1864186965d51d9738877b8: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260618 23:28:45.136525 22099 tablet_bootstrap.cc:654] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca: Neither blocks nor log segments found. Creating new log.
I20260618 23:28:45.137090 22099 log.cc:824] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca: Log is configured to *not* fsync() on all Append() calls
I20260618 23:28:45.137364 22094 raft_consensus.cc:348] T 2d22d1d282d24875a54526b044ab0958 P d5b08fbbb1864186965d51d9738877b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e617757a0b49e4961839bae4389d7c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45297 } } peers { permanent_uuid: "fbd16d57fbff4156b4549511ed9319ca" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46529 } } peers { permanent_uuid: "d5b08fbbb1864186965d51d9738877b8" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 33769 } }
I20260618 23:28:45.137532 22094 raft_consensus.cc:374] T 2d22d1d282d24875a54526b044ab0958 P d5b08fbbb1864186965d51d9738877b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:28:45.137578 22094 raft_consensus.cc:729] T 2d22d1d282d24875a54526b044ab0958 P d5b08fbbb1864186965d51d9738877b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b08fbbb1864186965d51d9738877b8, State: Initialized, Role: FOLLOWER
I20260618 23:28:45.137692 22094 consensus_queue.cc:260] T 2d22d1d282d24875a54526b044ab0958 P d5b08fbbb1864186965d51d9738877b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e617757a0b49e4961839bae4389d7c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45297 } } peers { permanent_uuid: "fbd16d57fbff4156b4549511ed9319ca" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46529 } } peers { permanent_uuid: "d5b08fbbb1864186965d51d9738877b8" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 33769 } }
I20260618 23:28:45.137967 22094 ts_tablet_manager.cc:1428] T 2d22d1d282d24875a54526b044ab0958 P d5b08fbbb1864186965d51d9738877b8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260618 23:28:45.138015 22077 heartbeater.cc:503] Master 127.8.39.190:41745 was elected leader, sending a full tablet report...
I20260618 23:28:45.138049 22099 tablet_bootstrap.cc:492] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca: No bootstrap required, opened a new log
I20260618 23:28:45.138114 22099 ts_tablet_manager.cc:1397] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 23:28:45.140094 22099 raft_consensus.cc:348] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e617757a0b49e4961839bae4389d7c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45297 } } peers { permanent_uuid: "fbd16d57fbff4156b4549511ed9319ca" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46529 } } peers { permanent_uuid: "d5b08fbbb1864186965d51d9738877b8" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 33769 } }
I20260618 23:28:45.140233 22099 raft_consensus.cc:374] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:28:45.140259 22099 raft_consensus.cc:729] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbd16d57fbff4156b4549511ed9319ca, State: Initialized, Role: FOLLOWER
I20260618 23:28:45.140377 22099 consensus_queue.cc:260] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e617757a0b49e4961839bae4389d7c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45297 } } peers { permanent_uuid: "fbd16d57fbff4156b4549511ed9319ca" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46529 } } peers { permanent_uuid: "d5b08fbbb1864186965d51d9738877b8" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 33769 } }
I20260618 23:28:45.140679 21943 heartbeater.cc:503] Master 127.8.39.190:41745 was elected leader, sending a full tablet report...
I20260618 23:28:45.140683 22099 ts_tablet_manager.cc:1428] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260618 23:28:45.158812 21944 tablet.cc:2369] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:28:45.214087 22102 raft_consensus.cc:482] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:28:45.214394 22102 raft_consensus.cc:504] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e617757a0b49e4961839bae4389d7c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45297 } } peers { permanent_uuid: "fbd16d57fbff4156b4549511ed9319ca" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46529 } } peers { permanent_uuid: "d5b08fbbb1864186965d51d9738877b8" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 33769 } }
I20260618 23:28:45.214879 22102 leader_election.cc:288] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53e617757a0b49e4961839bae4389d7c (127.8.39.129:45297), d5b08fbbb1864186965d51d9738877b8 (127.8.39.131:33769)
I20260618 23:28:45.218355 22091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:45.215323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:53055 (local address 127.8.39.129:45297)
0618 23:28:45.215394 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:45.215396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:45.215405 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:28:45.215435 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:45.215436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:45.215450 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:45.215515 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:28:45.215780 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.216453 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.217445 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.217942 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.218053 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:45.218064 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:45.218244 (+ 180us) server_negotiation.cc:299] Negotiation successful
0618 23:28:45.218282 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 23:28:45.218343 21945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:45.215149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:45297 (local address 127.8.39.130:53055)
0618 23:28:45.215246 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:45.215260 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:45.215326 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:45.215544 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:45.215546 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:45.215548 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:28:45.215658 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:45.215660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.216532 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.216533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:45.217363 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:45.217365 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.218064 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.218065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:45.218145 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:45.218150 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:28:45.218213 (+ 63us) client_negotiation.cc:240] Negotiation successful
0618 23:28:45.218260 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":4}
I20260618 23:28:45.218523 22103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:45.215267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:33769 (local address 127.8.39.130:46741)
0618 23:28:45.215477 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:45.215486 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:45.215524 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:45.215762 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:45.215763 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:45.215766 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:28:45.215842 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:45.215844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.216681 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.216682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:45.217639 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:45.217641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.218265 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.218266 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:45.218334 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:45.218337 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:28:45.218398 (+ 61us) client_negotiation.cc:240] Negotiation successful
0618 23:28:45.218441 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":116,"threads_started":1}
I20260618 23:28:45.218516 22093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:45.215483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:46741 (local address 127.8.39.131:33769)
0618 23:28:45.215553 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:45.215555 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:45.215576 (+ 21us) server_negotiation.cc:407] Connection header received
0618 23:28:45.215601 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:45.215601 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:45.215616 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:45.215683 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:28:45.215930 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.216573 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.217731 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.218191 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.218255 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:45.218263 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:45.218421 (+ 158us) server_negotiation.cc:299] Negotiation successful
0618 23:28:45.218463 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260618 23:28:45.218942 21763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d22d1d282d24875a54526b044ab0958" candidate_uuid: "fbd16d57fbff4156b4549511ed9319ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e617757a0b49e4961839bae4389d7c" is_pre_election: true
I20260618 23:28:45.218945 22031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d22d1d282d24875a54526b044ab0958" candidate_uuid: "fbd16d57fbff4156b4549511ed9319ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b08fbbb1864186965d51d9738877b8" is_pre_election: true
I20260618 23:28:45.219102 21763 raft_consensus.cc:2454] T 2d22d1d282d24875a54526b044ab0958 P 53e617757a0b49e4961839bae4389d7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbd16d57fbff4156b4549511ed9319ca in term 0.
I20260618 23:28:45.219103 22031 raft_consensus.cc:2454] T 2d22d1d282d24875a54526b044ab0958 P d5b08fbbb1864186965d51d9738877b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fbd16d57fbff4156b4549511ed9319ca in term 0.
I20260618 23:28:45.219344 21833 leader_election.cc:302] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5b08fbbb1864186965d51d9738877b8, fbd16d57fbff4156b4549511ed9319ca; no voters:
I20260618 23:28:45.219475 22102 raft_consensus.cc:2781] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:28:45.219558 22102 raft_consensus.cc:482] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:28:45.219595 22102 raft_consensus.cc:3037] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:28:45.220319 22102 raft_consensus.cc:504] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e617757a0b49e4961839bae4389d7c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45297 } } peers { permanent_uuid: "fbd16d57fbff4156b4549511ed9319ca" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46529 } } peers { permanent_uuid: "d5b08fbbb1864186965d51d9738877b8" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 33769 } }
I20260618 23:28:45.220469 22102 leader_election.cc:288] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca [CANDIDATE]: Term 1 election: Requested vote from peers 53e617757a0b49e4961839bae4389d7c (127.8.39.129:45297), d5b08fbbb1864186965d51d9738877b8 (127.8.39.131:33769)
I20260618 23:28:45.220695 21763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d22d1d282d24875a54526b044ab0958" candidate_uuid: "fbd16d57fbff4156b4549511ed9319ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e617757a0b49e4961839bae4389d7c"
I20260618 23:28:45.220759 21763 raft_consensus.cc:3037] T 2d22d1d282d24875a54526b044ab0958 P 53e617757a0b49e4961839bae4389d7c [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:28:45.221520 21763 raft_consensus.cc:2454] T 2d22d1d282d24875a54526b044ab0958 P 53e617757a0b49e4961839bae4389d7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbd16d57fbff4156b4549511ed9319ca in term 1.
I20260618 23:28:45.221607 22031 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2d22d1d282d24875a54526b044ab0958" candidate_uuid: "fbd16d57fbff4156b4549511ed9319ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5b08fbbb1864186965d51d9738877b8"
I20260618 23:28:45.221697 21833 leader_election.cc:302] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e617757a0b49e4961839bae4389d7c, fbd16d57fbff4156b4549511ed9319ca; no voters:
I20260618 23:28:45.221735 22031 raft_consensus.cc:3037] T 2d22d1d282d24875a54526b044ab0958 P d5b08fbbb1864186965d51d9738877b8 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:28:45.221823 22102 raft_consensus.cc:2781] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:28:45.221993 22102 raft_consensus.cc:686] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca [term 1 LEADER]: Becoming Leader. State: Replica: fbd16d57fbff4156b4549511ed9319ca, State: Running, Role: LEADER
I20260618 23:28:45.222087 22102 consensus_queue.cc:237] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e617757a0b49e4961839bae4389d7c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45297 } } peers { permanent_uuid: "fbd16d57fbff4156b4549511ed9319ca" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46529 } } peers { permanent_uuid: "d5b08fbbb1864186965d51d9738877b8" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 33769 } }
I20260618 23:28:45.222807 22031 raft_consensus.cc:2454] T 2d22d1d282d24875a54526b044ab0958 P d5b08fbbb1864186965d51d9738877b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fbd16d57fbff4156b4549511ed9319ca in term 1.
I20260618 23:28:45.223153 21618 catalog_manager.cc:5495] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca reported cstate change: term changed from 0 to 1, leader changed from <none> to fbd16d57fbff4156b4549511ed9319ca (127.8.39.130). New cstate: current_term: 1 leader_uuid: "fbd16d57fbff4156b4549511ed9319ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e617757a0b49e4961839bae4389d7c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45297 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fbd16d57fbff4156b4549511ed9319ca" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46529 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5b08fbbb1864186965d51d9738877b8" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 33769 } health_report { overall_health: UNKNOWN } } }
I20260618 23:28:45.244194 22090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:45.241875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42580 (local address 127.8.39.130:46529)
0618 23:28:45.241919 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:45.241921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:45.241935 (+ 14us) server_negotiation.cc:407] Connection header received
0618 23:28:45.241971 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:45.241971 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:45.241990 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:45.242059 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:28:45.242221 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.242685 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.243440 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.243650 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.243710 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:45.243888 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:28:45.243988 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:28:45.244063 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:45.244099 (+ 36us) server_negotiation.cc:299] Negotiation successful
0618 23:28:45.244117 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260618 23:28:45.246413 22031 raft_consensus.cc:1261] T 2d22d1d282d24875a54526b044ab0958 P d5b08fbbb1864186965d51d9738877b8 [term 1 FOLLOWER]: Refusing update from remote peer fbd16d57fbff4156b4549511ed9319ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:28:45.246516 21763 raft_consensus.cc:1261] T 2d22d1d282d24875a54526b044ab0958 P 53e617757a0b49e4961839bae4389d7c [term 1 FOLLOWER]: Refusing update from remote peer fbd16d57fbff4156b4549511ed9319ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:28:45.246831 22104 consensus_queue.cc:1035] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5b08fbbb1864186965d51d9738877b8" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 33769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:28:45.246963 22102 consensus_queue.cc:1035] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e617757a0b49e4961839bae4389d7c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260618 23:28:45.248569 21810 tablet.cc:2369] T 2d22d1d282d24875a54526b044ab0958 P 53e617757a0b49e4961839bae4389d7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:28:45.250275 22108 mvcc.cc:205] Tried to move back new op lower bound from 7298356532206133248 to 7298356532109840384. Current Snapshot: MvccSnapshot[applied={T|T < 7298356532206133248 or (T in {7298356532206133248})}]
I20260618 23:28:45.252039 21617 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:28:45.252167 21617 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56590:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:28:45.253093 21617 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca: Sending DeleteTablet for 3 replicas of tablet 2d22d1d282d24875a54526b044ab0958
I20260618 23:28:45.253482 21877 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d22d1d282d24875a54526b044ab0958 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:28:45 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:42564
I20260618 23:28:45.253520 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 21680
I20260618 23:28:45.253708 21743 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d22d1d282d24875a54526b044ab0958 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:28:45 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:46210
I20260618 23:28:45.254035 22011 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d22d1d282d24875a54526b044ab0958 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:28:45 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:59204
I20260618 23:28:45.254385 22117 tablet_replica.cc:330] T 2d22d1d282d24875a54526b044ab0958 P d5b08fbbb1864186965d51d9738877b8: stopping tablet replica
I20260618 23:28:45.254483 22117 raft_consensus.cc:2229] T 2d22d1d282d24875a54526b044ab0958 P d5b08fbbb1864186965d51d9738877b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:28:45.254583 22117 raft_consensus.cc:2258] T 2d22d1d282d24875a54526b044ab0958 P d5b08fbbb1864186965d51d9738877b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:28:45.255323 22117 ts_tablet_manager.cc:1905] T 2d22d1d282d24875a54526b044ab0958 P d5b08fbbb1864186965d51d9738877b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:28:45.257364 22117 ts_tablet_manager.cc:1918] T 2d22d1d282d24875a54526b044ab0958 P d5b08fbbb1864186965d51d9738877b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:28:45.257445 22117 log.cc:1196] T 2d22d1d282d24875a54526b044ab0958 P d5b08fbbb1864186965d51d9738877b8: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/wal/wals/2d22d1d282d24875a54526b044ab0958
I20260618 23:28:45.257730 22117 ts_tablet_manager.cc:1939] T 2d22d1d282d24875a54526b044ab0958 P d5b08fbbb1864186965d51d9738877b8: Deleting consensus metadata
I20260618 23:28:45.258206 21605 catalog_manager.cc:4841] TS d5b08fbbb1864186965d51d9738877b8 (127.8.39.131:33769): tablet 2d22d1d282d24875a54526b044ab0958 (table test-table [id=8ec18b477ff04671b5909b34f2d06546]) successfully deleted
W20260618 23:28:45.259660 21605 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.39.129:45297 (error 108)
I20260618 23:28:45.260070 22116 tablet_replica.cc:330] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca: stopping tablet replica
I20260618 23:28:45.260197 22116 raft_consensus.cc:2229] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:28:45.260228 22089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:45.260081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:45297 (local address 127.0.0.1:46222)
0618 23:28:45.260146 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:45.260193 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.129:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260618 23:28:45.260303 22116 raft_consensus.cc:2258] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca [term 1 FOLLOWER]: Raft consensus is shut down!
W20260618 23:28:45.260360 21605 catalog_manager.cc:4584] TS 53e617757a0b49e4961839bae4389d7c (127.8.39.129:45297): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2d22d1d282d24875a54526b044ab0958: Network error: Client connection negotiation failed: client connection to 127.8.39.129:45297: connect: Connection refused (error 111)
I20260618 23:28:45.260511 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 21814
I20260618 23:28:45.265200 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 21948
I20260618 23:28:45.265353 22089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:45.265216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:46529 (local address 127.0.0.1:42584)
0618 23:28:45.265278 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:45.265312 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.130:46529: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260618 23:28:45.269853 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 21586
I20260618 23:28:45.275038 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:41745
--webserver_interface=127.8.39.190
--webserver_port=46293
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:41745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:28:45.358198 22119 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:28:45.358412 22119 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:28:45.358436 22119 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:28:45.359535 22119 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:28:45.359572 22119 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:28:45.359586 22119 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:28:45.359598 22119 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:28:45.359611 22119 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:28:45.361011 22119 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:41745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:41745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=46293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22119
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:28:45.361316 22119 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:28:45.361503 22119 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:28:45.370052 22126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:45.370070 22128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:45.370092 22125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:28:45.370370 22119 server_base.cc:1034] running on GCE node
I20260618 23:28:45.370692 22119 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:28:45.371125 22119 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:28:45.372325 22119 hybrid_clock.cc:648] HybridClock initialized: now 1781825325372298 us; error 43 us; skew 500 ppm
Jun 18 23:28:45 dist-test-slave-47qt krb5kdc[21573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825325, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:28:45.380122 22119 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:28:45.380607 22119 webserver.cc:460] Webserver started at http://127.8.39.190:46293/ using document root <none> and password file <none>
I20260618 23:28:45.380878 22119 fs_manager.cc:359] Metadata directory not provided
I20260618 23:28:45.380934 22119 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:28:45.382500 22119 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:28:45.383286 22135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:45.383502 22119 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:28:45.383620 22119 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "b0ea7459b1874d59b31a19576c346dca"
format_stamp: "Formatted at 2026-06-18 23:28:44 on dist-test-slave-47qt"
server_key: "437c6fcb71deec0cbafc9324a92fcb6f"
server_key_iv: "30288963de06b5eae6b057536d2e63f1"
server_key_version: "encryptionkey@0"
I20260618 23:28:45.384053 22119 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:28:45.412374 22119 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:28:45.415158 22119 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:28:45.415350 22119 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:28:45.419600 22119 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:41745
I20260618 23:28:45.419633 22187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:41745 every 8 connection(s)
I20260618 23:28:45.420069 22119 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:28:45.421816 22188 sys_catalog.cc:263] Verifying existing consensus state
I20260618 23:28:45.422504 22188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca: Bootstrap starting.
I20260618 23:28:45.430186 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 22119
I20260618 23:28:45.430440 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.129:45297
--local_ip_for_outbound_sockets=127.8.39.129
--tserver_master_addrs=127.8.39.190:41745
--webserver_port=33481
--webserver_interface=127.8.39.129
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:28:45.432374 22188 log.cc:824] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca: Log is configured to *not* fsync() on all Append() calls
I20260618 23:28:45.435586 22188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 23:28:45.435983 22188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca: Bootstrap complete.
I20260618 23:28:45.440135 22188 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ea7459b1874d59b31a19576c346dca" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41745 } }
I20260618 23:28:45.440533 22188 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ea7459b1874d59b31a19576c346dca, State: Initialized, Role: FOLLOWER
I20260618 23:28:45.440730 22188 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ea7459b1874d59b31a19576c346dca" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41745 } }
I20260618 23:28:45.440821 22188 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:28:45.440876 22188 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:28:45.440927 22188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [term 1 FOLLOWER]: Advancing to term 2
I20260618 23:28:45.442508 22188 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ea7459b1874d59b31a19576c346dca" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41745 } }
I20260618 23:28:45.442634 22188 leader_election.cc:302] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0ea7459b1874d59b31a19576c346dca; no voters:
I20260618 23:28:45.442802 22188 leader_election.cc:288] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [CANDIDATE]: Term 2 election: Requested vote from peers
I20260618 23:28:45.442880 22192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [term 2 FOLLOWER]: Leader election won for term 2
I20260618 23:28:45.443039 22188 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:28:45.443045 22192 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [term 2 LEADER]: Becoming Leader. State: Replica: b0ea7459b1874d59b31a19576c346dca, State: Running, Role: LEADER
I20260618 23:28:45.443255 22192 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ea7459b1874d59b31a19576c346dca" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41745 } }
I20260618 23:28:45.443538 22192 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0ea7459b1874d59b31a19576c346dca. Latest consensus state: current_term: 2 leader_uuid: "b0ea7459b1874d59b31a19576c346dca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ea7459b1874d59b31a19576c346dca" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41745 } } }
I20260618 23:28:45.443638 22192 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [sys.catalog]: This master's current role is: LEADER
I20260618 23:28:45.443539 22193 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "b0ea7459b1874d59b31a19576c346dca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ea7459b1874d59b31a19576c346dca" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41745 } } }
I20260618 23:28:45.443748 22193 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [sys.catalog]: This master's current role is: LEADER
I20260618 23:28:45.443848 22201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:28:45.444623 22201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:28:45.444744 22201 catalog_manager.cc:1259] Loaded cluster ID: 4c8069eb6e0a42d4a48c413ff14e0888
I20260618 23:28:45.444782 22201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:28:45.445232 22201 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260618 23:28:45.450071 22201 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260618 23:28:45.452689 22201 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:28:45.452917 22201 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca: Loaded TSK: 0
I20260618 23:28:45.453176 22201 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260618 23:28:45.517663 22190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:28:45.517915 22190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:28:45.517954 22190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:28:45.519160 22190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:28:45.519234 22190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:28:45.519287 22190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.129
I20260618 23:28:45.520795 22190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.129:45297
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.39.129
--webserver_port=33481
--tserver_master_addrs=127.8.39.190:41745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.129
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:28:45.521060 22190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:28:45.521353 22190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 23:28:45.527817 22190 server_base.cc:1034] running on GCE node
W20260618 23:28:45.527827 22216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:45.527928 22215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:45.527827 22218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:28:45.528245 22190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:28:45.528508 22190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:28:45.529706 22190 hybrid_clock.cc:648] HybridClock initialized: now 1781825325529689 us; error 31 us; skew 500 ppm
Jun 18 23:28:45 dist-test-slave-47qt krb5kdc[21573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825325, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:28:45.532780 22190 init.cc:377] Logged in from keytab as kudu/127.8.39.129@KRBTEST.COM (short username kudu)
I20260618 23:28:45.533339 22190 webserver.cc:460] Webserver started at http://127.8.39.129:33481/ using document root <none> and password file <none>
I20260618 23:28:45.533583 22190 fs_manager.cc:359] Metadata directory not provided
I20260618 23:28:45.533668 22190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:28:45.535061 22190 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:28:45.535916 22225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:45.536173 22190 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:28:45.536265 22190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/wal
uuid: "53e617757a0b49e4961839bae4389d7c"
format_stamp: "Formatted at 2026-06-18 23:28:44 on dist-test-slave-47qt"
server_key: "b1003fcb2581ac6d36e977d9574190b9"
server_key_iv: "b7f9506bc75ebb0a1ac891376cbdda1b"
server_key_version: "encryptionkey@0"
I20260618 23:28:45.536669 22190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:28:45.548864 22190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:28:45.551360 22190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:28:45.551506 22190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:28:45.551761 22190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:28:45.552277 22232 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260618 23:28:45.553857 22190 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260618 23:28:45.553917 22190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260618 23:28:45.553944 22190 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260618 23:28:45.554721 22190 ts_tablet_manager.cc:610] Registered 1 tablets
I20260618 23:28:45.554769 22190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260618 23:28:45.554814 22232 tablet_bootstrap.cc:492] T 2d22d1d282d24875a54526b044ab0958 P 53e617757a0b49e4961839bae4389d7c: Bootstrap starting.
I20260618 23:28:45.562695 22190 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.129:45297
I20260618 23:28:45.562927 22339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.129:45297 every 8 connection(s)
I20260618 23:28:45.563230 22190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
I20260618 23:28:45.566138 22232 log.cc:824] T 2d22d1d282d24875a54526b044ab0958 P 53e617757a0b49e4961839bae4389d7c: Log is configured to *not* fsync() on all Append() calls
I20260618 23:28:45.566306 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 22190
I20260618 23:28:45.566599 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.130:46529
--local_ip_for_outbound_sockets=127.8.39.130
--tserver_master_addrs=127.8.39.190:41745
--webserver_port=43549
--webserver_interface=127.8.39.130
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:28:45.568679 22232 tablet_bootstrap.cc:492] T 2d22d1d282d24875a54526b044ab0958 P 53e617757a0b49e4961839bae4389d7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 23:28:45.568977 22232 tablet_bootstrap.cc:492] T 2d22d1d282d24875a54526b044ab0958 P 53e617757a0b49e4961839bae4389d7c: Bootstrap complete.
I20260618 23:28:45.570221 22232 ts_tablet_manager.cc:1397] T 2d22d1d282d24875a54526b044ab0958 P 53e617757a0b49e4961839bae4389d7c: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.009s
I20260618 23:28:45.571436 22232 raft_consensus.cc:348] T 2d22d1d282d24875a54526b044ab0958 P 53e617757a0b49e4961839bae4389d7c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e617757a0b49e4961839bae4389d7c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45297 } } peers { permanent_uuid: "fbd16d57fbff4156b4549511ed9319ca" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46529 } } peers { permanent_uuid: "d5b08fbbb1864186965d51d9738877b8" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 33769 } }
I20260618 23:28:45.571686 22232 raft_consensus.cc:729] T 2d22d1d282d24875a54526b044ab0958 P 53e617757a0b49e4961839bae4389d7c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e617757a0b49e4961839bae4389d7c, State: Initialized, Role: FOLLOWER
I20260618 23:28:45.571854 22232 consensus_queue.cc:260] T 2d22d1d282d24875a54526b044ab0958 P 53e617757a0b49e4961839bae4389d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e617757a0b49e4961839bae4389d7c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45297 } } peers { permanent_uuid: "fbd16d57fbff4156b4549511ed9319ca" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46529 } } peers { permanent_uuid: "d5b08fbbb1864186965d51d9738877b8" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 33769 } }
I20260618 23:28:45.572144 22232 ts_tablet_manager.cc:1428] T 2d22d1d282d24875a54526b044ab0958 P 53e617757a0b49e4961839bae4389d7c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
Jun 18 23:28:45 dist-test-slave-47qt krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825325, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:28:45.578606 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:45.563912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.129:55105)
0618 23:28:45.564192 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:45.564234 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:45.564831 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:45.565644 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:45.565649 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:45.566041 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:28:45.566306 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:45.566313 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.567948 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.567950 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:45.569253 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:45.569256 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.570112 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.570113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:45.570262 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:45.573341 (+ 3079us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:28:45.573349 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:28:45.574773 (+ 1424us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:28:45.576772 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:45.576779 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:45.576781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:45.577017 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:45.577443 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:45.577444 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:45.577444 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:45.577562 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:45.577957 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:28:45.577964 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:28:45.578103 (+ 139us) client_negotiation.cc:769] Sending connection context
0618 23:28:45.578215 (+ 112us) client_negotiation.cc:240] Negotiation successful
0618 23:28:45.578407 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"thread_start_us":88,"threads_started":1}
I20260618 23:28:45.578642 22343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:45.564032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:55105 (local address 127.8.39.190:41745)
0618 23:28:45.565191 (+ 1159us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:45.565200 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:45.565237 (+ 37us) server_negotiation.cc:407] Connection header received
0618 23:28:45.565368 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:45.565375 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:45.565428 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:45.565568 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:45.566452 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.567820 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.569346 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.570010 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.570118 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:45.574900 (+ 4782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:45.574907 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:45.574907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:45.574923 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:45.576656 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:45.577106 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:45.577109 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:45.577110 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:45.577330 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:45.577638 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:45.577639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:45.577639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:45.577877 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:45.577978 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:45.578267 (+ 289us) server_negotiation.cc:299] Negotiation successful
0618 23:28:45.578407 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1109,"thread_start_us":86,"threads_started":1}
I20260618 23:28:45.579909 22340 heartbeater.cc:347] Connected to a master server at 127.8.39.190:41745
I20260618 23:28:45.580015 22340 heartbeater.cc:464] Registering TS with master...
I20260618 23:28:45.580190 22340 heartbeater.cc:511] Master 127.8.39.190:41745 requested a full tablet report, sending...
I20260618 23:28:45.581014 22152 ts_manager.cc:194] Registered new tserver with Master: 53e617757a0b49e4961839bae4389d7c (127.8.39.129:45297)
I20260618 23:28:45.582231 22152 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.129@KRBTEST.COM'} at 127.8.39.129:55105
I20260618 23:28:45.583196 22340 heartbeater.cc:503] Master 127.8.39.190:41745 was elected leader, sending a full tablet report...
Jun 18 23:28:45 dist-test-slave-47qt krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825325, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for kudu/127.8.39.129@KRBTEST.COM
I20260618 23:28:45.591200 22348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:45.581887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46226 (local address 127.8.39.129:45297)
0618 23:28:45.582243 (+ 356us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:45.582247 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:45.582256 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:28:45.582289 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:45.582291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:45.582399 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:45.582467 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:45.583210 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.584138 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.584912 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.585299 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.585399 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:45.587592 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:45.587600 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:45.587601 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:45.587608 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:45.589429 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:45.590246 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:45.590249 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:45.590249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:45.590325 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:45.590549 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:45.590551 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:45.590551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:45.590728 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:45.590824 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:45.591004 (+ 180us) server_negotiation.cc:299] Negotiation successful
0618 23:28:45.591097 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":69,"threads_started":1}
I20260618 23:28:45.591212 22347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:45.581624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:45297 (local address 127.0.0.1:46226)
0618 23:28:45.582033 (+ 409us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:45.582066 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:45.582128 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:45.582731 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:45.582737 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:45.583051 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:28:45.583115 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:45.583118 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.584229 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.584230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:45.584807 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:45.584809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.585401 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.585402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:45.585483 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:45.586014 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:28:45.586018 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:28:45.587475 (+ 1457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:28:45.589533 (+ 2058us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:45.589539 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:45.589541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:45.589940 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:45.590401 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:45.590402 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:45.590403 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:45.590481 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:45.590803 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:28:45.590809 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:28:45.590863 (+ 54us) client_negotiation.cc:769] Sending connection context
0618 23:28:45.590968 (+ 105us) client_negotiation.cc:240] Negotiation successful
0618 23:28:45.591098 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":90,"threads_started":1}
I20260618 23:28:45.591966 22274 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d22d1d282d24875a54526b044ab0958 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:28:45 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:46226
I20260618 23:28:45.592478 22349 tablet_replica.cc:330] stopping tablet replica
I20260618 23:28:45.592624 22349 raft_consensus.cc:2229] T 2d22d1d282d24875a54526b044ab0958 P 53e617757a0b49e4961839bae4389d7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:28:45.592705 22349 raft_consensus.cc:2258] T 2d22d1d282d24875a54526b044ab0958 P 53e617757a0b49e4961839bae4389d7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:28:45.593014 22349 ts_tablet_manager.cc:1905] T 2d22d1d282d24875a54526b044ab0958 P 53e617757a0b49e4961839bae4389d7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:28:45.595113 22349 ts_tablet_manager.cc:1918] T 2d22d1d282d24875a54526b044ab0958 P 53e617757a0b49e4961839bae4389d7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:28:45.595234 22349 log.cc:1196] T 2d22d1d282d24875a54526b044ab0958 P 53e617757a0b49e4961839bae4389d7c: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/wal/wals/2d22d1d282d24875a54526b044ab0958
I20260618 23:28:45.595464 22349 ts_tablet_manager.cc:1939] T 2d22d1d282d24875a54526b044ab0958 P 53e617757a0b49e4961839bae4389d7c: Deleting consensus metadata
I20260618 23:28:45.595992 22139 catalog_manager.cc:4841] TS 53e617757a0b49e4961839bae4389d7c (127.8.39.129:45297): tablet 2d22d1d282d24875a54526b044ab0958 (table test-table [id=8ec18b477ff04671b5909b34f2d06546]) successfully deleted
W20260618 23:28:45.664621 22345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:28:45.664834 22345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:28:45.664857 22345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:28:45.666062 22345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:28:45.666105 22345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:28:45.666175 22345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.130
I20260618 23:28:45.667675 22345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.130:46529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.39.130
--webserver_port=43549
--tserver_master_addrs=127.8.39.190:41745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.130
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:28:45.667865 22345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:28:45.668041 22345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:28:45.673166 22357 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:45.673211 22354 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:45.673211 22355 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:28:45.673339 22345 server_base.cc:1034] running on GCE node
I20260618 23:28:45.673619 22345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:28:45.673892 22345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:28:45.675077 22345 hybrid_clock.cc:648] HybridClock initialized: now 1781825325675055 us; error 38 us; skew 500 ppm
Jun 18 23:28:45 dist-test-slave-47qt krb5kdc[21573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825325, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:28:45.677706 22345 init.cc:377] Logged in from keytab as kudu/127.8.39.130@KRBTEST.COM (short username kudu)
I20260618 23:28:45.678172 22345 webserver.cc:460] Webserver started at http://127.8.39.130:43549/ using document root <none> and password file <none>
I20260618 23:28:45.678433 22345 fs_manager.cc:359] Metadata directory not provided
I20260618 23:28:45.678515 22345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:28:45.679971 22345 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260618 23:28:45.680945 22364 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:45.681217 22345 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:28:45.681341 22345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/wal
uuid: "fbd16d57fbff4156b4549511ed9319ca"
format_stamp: "Formatted at 2026-06-18 23:28:44 on dist-test-slave-47qt"
server_key: "3fe41cd30ff79803d61fc86ca00f8291"
server_key_iv: "15032fbbeac4574ce0161e3b2f178cd7"
server_key_version: "encryptionkey@0"
I20260618 23:28:45.681775 22345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:28:45.706382 22345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:28:45.708691 22345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:28:45.708981 22345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:28:45.709277 22345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:28:45.709736 22371 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260618 23:28:45.711306 22345 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260618 23:28:45.711410 22345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260618 23:28:45.711472 22345 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260618 23:28:45.712212 22345 ts_tablet_manager.cc:610] Registered 1 tablets
I20260618 23:28:45.712297 22345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260618 23:28:45.712360 22371 tablet_bootstrap.cc:492] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca: Bootstrap starting.
I20260618 23:28:45.718343 22345 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.130:46529
I20260618 23:28:45.718426 22478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.130:46529 every 8 connection(s)
I20260618 23:28:45.718756 22345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
I20260618 23:28:45.722448 22371 log.cc:824] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca: Log is configured to *not* fsync() on all Append() calls
Jun 18 23:28:45 dist-test-slave-47qt krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825325, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:28:45.724866 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 22345
I20260618 23:28:45.725188 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.131:33769
--local_ip_for_outbound_sockets=127.8.39.131
--tserver_master_addrs=127.8.39.190:41745
--webserver_port=40087
--webserver_interface=127.8.39.131
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:28:45.725713 22371 tablet_bootstrap.cc:492] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 23:28:45.726087 22371 tablet_bootstrap.cc:492] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca: Bootstrap complete.
I20260618 23:28:45.727113 22371 ts_tablet_manager.cc:1397] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.008s
I20260618 23:28:45.728379 22371 raft_consensus.cc:348] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e617757a0b49e4961839bae4389d7c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45297 } } peers { permanent_uuid: "fbd16d57fbff4156b4549511ed9319ca" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46529 } } peers { permanent_uuid: "d5b08fbbb1864186965d51d9738877b8" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 33769 } }
I20260618 23:28:45.728693 22371 raft_consensus.cc:729] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbd16d57fbff4156b4549511ed9319ca, State: Initialized, Role: FOLLOWER
I20260618 23:28:45.728744 22343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:45.719599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:36617 (local address 127.8.39.190:41745)
0618 23:28:45.719659 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:45.719665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:45.719886 (+ 221us) server_negotiation.cc:407] Connection header received
0618 23:28:45.720417 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:45.720418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:45.720429 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:45.720487 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:45.721098 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.721649 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.722697 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.723028 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.723121 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:45.725566 (+ 2445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:45.725568 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:45.725568 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:45.725577 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:45.727078 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:45.727610 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:45.727611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:45.727612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:45.727678 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:45.727983 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:45.727984 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:45.727984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:45.728131 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:45.728208 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:45.728544 (+ 336us) server_negotiation.cc:299] Negotiation successful
0618 23:28:45.728638 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260618 23:28:45.728843 22371 consensus_queue.cc:260] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "53e617757a0b49e4961839bae4389d7c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45297 } } peers { permanent_uuid: "fbd16d57fbff4156b4549511ed9319ca" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46529 } } peers { permanent_uuid: "d5b08fbbb1864186965d51d9738877b8" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 33769 } }
I20260618 23:28:45.729192 22371 ts_tablet_manager.cc:1428] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20260618 23:28:45.729225 22481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:45.719550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.130:36617)
0618 23:28:45.719763 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:45.719800 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:45.720306 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:45.720542 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:45.720548 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:45.720811 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:28:45.720997 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:45.721002 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.721786 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.721787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:45.722589 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:45.722591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.723163 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.723164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:45.723279 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:45.723901 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:28:45.723909 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:28:45.725457 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:28:45.727184 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:45.727192 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:45.727194 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:45.727521 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:45.727765 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:45.727766 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:45.727766 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:45.727905 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:45.728208 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:28:45.728214 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:28:45.728386 (+ 172us) client_negotiation.cc:769] Sending connection context
0618 23:28:45.728495 (+ 109us) client_negotiation.cc:240] Negotiation successful
0618 23:28:45.728675 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":107,"threads_started":1}
I20260618 23:28:45.729674 22479 heartbeater.cc:347] Connected to a master server at 127.8.39.190:41745
I20260618 23:28:45.729773 22479 heartbeater.cc:464] Registering TS with master...
I20260618 23:28:45.729945 22479 heartbeater.cc:511] Master 127.8.39.190:41745 requested a full tablet report, sending...
I20260618 23:28:45.730558 22152 ts_manager.cc:194] Registered new tserver with Master: fbd16d57fbff4156b4549511ed9319ca (127.8.39.130:46529)
I20260618 23:28:45.731213 22152 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.130@KRBTEST.COM'} at 127.8.39.130:36617
I20260618 23:28:45.732134 22479 heartbeater.cc:503] Master 127.8.39.190:41745 was elected leader, sending a full tablet report...
Jun 18 23:28:45 dist-test-slave-47qt krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825325, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for kudu/127.8.39.130@KRBTEST.COM
I20260618 23:28:45.740453 22347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:45.731467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:46529 (local address 127.0.0.1:42586)
0618 23:28:45.731516 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:45.731527 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:45.731565 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:45.731908 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:45.731910 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:45.732101 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:28:45.732157 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:45.732159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.733561 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.733562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:45.734188 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:45.734191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.735244 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.735245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:45.735299 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:45.735819 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:28:45.735824 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:28:45.737313 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:28:45.739351 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:45.739352 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:45.739352 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:45.739518 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:45.739742 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:45.739743 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:45.739744 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:45.739817 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:45.740092 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:28:45.740093 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:28:45.740129 (+ 36us) client_negotiation.cc:769] Sending connection context
0618 23:28:45.740233 (+ 104us) client_negotiation.cc:240] Negotiation successful
0618 23:28:45.740366 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260618 23:28:45.740469 22485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:45.731536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42586 (local address 127.8.39.130:46529)
0618 23:28:45.731686 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:45.731690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:45.731700 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:28:45.731743 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:45.731745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:45.731758 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:45.731834 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:45.732276 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.733457 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.734353 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.735144 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.735238 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:45.737443 (+ 2205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:45.737452 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:45.737452 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:45.737460 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:45.739252 (+ 1792us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:45.739595 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:45.739598 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:45.739598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:45.739661 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:45.739867 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:45.739868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:45.739868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:45.740026 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:45.740107 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:45.740282 (+ 175us) server_negotiation.cc:299] Negotiation successful
0618 23:28:45.740372 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":78,"threads_started":1}
I20260618 23:28:45.740885 22413 tablet_service.cc:1514] Processing DeleteTablet for tablet 2d22d1d282d24875a54526b044ab0958 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:28:45 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:42586
I20260618 23:28:45.741509 22486 tablet_replica.cc:330] stopping tablet replica
I20260618 23:28:45.741616 22486 raft_consensus.cc:2229] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:28:45.741684 22486 raft_consensus.cc:2258] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:28:45.742112 22486 ts_tablet_manager.cc:1905] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:28:45.744520 22486 ts_tablet_manager.cc:1918] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:28:45.744596 22486 log.cc:1196] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/wal/wals/2d22d1d282d24875a54526b044ab0958
I20260618 23:28:45.745210 22486 ts_tablet_manager.cc:1939] T 2d22d1d282d24875a54526b044ab0958 P fbd16d57fbff4156b4549511ed9319ca: Deleting consensus metadata
I20260618 23:28:45.746060 22137 catalog_manager.cc:4841] TS fbd16d57fbff4156b4549511ed9319ca (127.8.39.130:46529): tablet 2d22d1d282d24875a54526b044ab0958 (table test-table [id=8ec18b477ff04671b5909b34f2d06546]) successfully deleted
W20260618 23:28:45.817502 22483 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:28:45.817775 22483 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:28:45.817827 22483 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:28:45.819173 22483 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:28:45.819243 22483 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:28:45.819290 22483 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.131
I20260618 23:28:45.820830 22483 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.131:33769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.39.131
--webserver_port=40087
--tserver_master_addrs=127.8.39.190:41745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22483
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.131
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:28:45.821103 22483 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:28:45.821367 22483 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 23:28:45.826933 22483 server_base.cc:1034] running on GCE node
W20260618 23:28:45.826956 22494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:45.826987 22492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:45.827195 22491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:28:45.827389 22483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:28:45.827594 22483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:28:45.828787 22483 hybrid_clock.cc:648] HybridClock initialized: now 1781825325828765 us; error 47 us; skew 500 ppm
Jun 18 23:28:45 dist-test-slave-47qt krb5kdc[21573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825325, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:28:45.831595 22483 init.cc:377] Logged in from keytab as kudu/127.8.39.131@KRBTEST.COM (short username kudu)
I20260618 23:28:45.832201 22483 webserver.cc:460] Webserver started at http://127.8.39.131:40087/ using document root <none> and password file <none>
I20260618 23:28:45.832489 22483 fs_manager.cc:359] Metadata directory not provided
I20260618 23:28:45.832571 22483 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:28:45.834793 22483 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:28:45.835786 22501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:45.835979 22483 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:28:45.836077 22483 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/wal
uuid: "d5b08fbbb1864186965d51d9738877b8"
format_stamp: "Formatted at 2026-06-18 23:28:45 on dist-test-slave-47qt"
server_key: "4d16de477a96d86f015d8a1824dbe10f"
server_key_iv: "5936a638752d8e0b2db368433c376bff"
server_key_version: "encryptionkey@0"
I20260618 23:28:45.836447 22483 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:28:45.854990 22483 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:28:45.857445 22483 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:28:45.857600 22483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:28:45.857843 22483 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:28:45.858199 22483 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:28:45.858232 22483 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:45.858255 22483 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:28:45.858299 22483 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:45.863890 22483 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.131:33769
I20260618 23:28:45.864018 22614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.131:33769 every 8 connection(s)
I20260618 23:28:45.864289 22483 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
Jun 18 23:28:45 dist-test-slave-47qt krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825325, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:28:45.872109 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 22483
I20260618 23:28:45.874346 22343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:45.865059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:34357 (local address 127.8.39.190:41745)
0618 23:28:45.865180 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:45.865182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:45.865328 (+ 146us) server_negotiation.cc:407] Connection header received
0618 23:28:45.865792 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:45.865793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:45.865803 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:45.865876 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:45.866969 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.867530 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.868828 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.869045 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.869168 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:45.871290 (+ 2122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:45.871292 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:45.871292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:45.871307 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:45.872877 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:45.873361 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:45.873362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:45.873362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:45.873418 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:45.873694 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:45.873695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:45.873695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:45.873809 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:45.873879 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:45.874172 (+ 293us) server_negotiation.cc:299] Negotiation successful
0618 23:28:45.874253 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260618 23:28:45.874531 22617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:45.865012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.131:34357)
0618 23:28:45.865200 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:45.865241 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:45.865711 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:45.865991 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:45.865995 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:45.866554 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:28:45.866737 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:45.866741 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.867703 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.867704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:45.868707 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:45.868710 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.869187 (+ 477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.869188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:45.869312 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:45.869842 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:28:45.869850 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:28:45.871191 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:28:45.872977 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:45.872984 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:45.872986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:45.873297 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:45.873501 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:45.873502 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:45.873502 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:45.873619 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:45.873882 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:28:45.873889 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:28:45.874035 (+ 146us) client_negotiation.cc:769] Sending connection context
0618 23:28:45.874137 (+ 102us) client_negotiation.cc:240] Negotiation successful
0618 23:28:45.874291 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":110,"threads_started":1}
I20260618 23:28:45.875269 22615 heartbeater.cc:347] Connected to a master server at 127.8.39.190:41745
I20260618 23:28:45.875373 22615 heartbeater.cc:464] Registering TS with master...
I20260618 23:28:45.875541 22615 heartbeater.cc:511] Master 127.8.39.190:41745 requested a full tablet report, sending...
I20260618 23:28:45.875941 22152 ts_manager.cc:194] Registered new tserver with Master: d5b08fbbb1864186965d51d9738877b8 (127.8.39.131:33769)
I20260618 23:28:45.876295 22152 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.131@KRBTEST.COM'} at 127.8.39.131:34357
I20260618 23:28:45.881038 22619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:45.872549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56604 (local address 127.8.39.190:41745)
0618 23:28:45.872668 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:45.872670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:45.872788 (+ 118us) server_negotiation.cc:407] Connection header received
0618 23:28:45.872864 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:45.872865 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:45.872898 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:45.872982 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:45.874222 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.875234 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.876193 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.876448 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.876505 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:45.877847 (+ 1342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:45.877848 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:45.877849 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:45.877855 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:45.879581 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:45.879982 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:45.879984 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:45.879984 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:45.880051 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:45.880271 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:45.880272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:45.880273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:45.880578 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:45.880774 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:45.880843 (+ 69us) server_negotiation.cc:299] Negotiation successful
0618 23:28:45.880932 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":69,"threads_started":1}
I20260618 23:28:45.881883 8350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:28:45.884718 22619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:45.882351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56606 (local address 127.8.39.190:41745)
0618 23:28:45.882388 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:45.882390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:45.882495 (+ 105us) server_negotiation.cc:407] Connection header received
0618 23:28:45.882551 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:45.882551 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:45.882574 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:45.882630 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:28:45.882779 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.883230 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.884047 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.884231 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.884286 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:45.884408 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:28:45.884494 (+ 86us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:28:45.884556 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:45.884608 (+ 52us) server_negotiation.cc:299] Negotiation successful
0618 23:28:45.884625 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260618 23:28:45.885186 22152 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:56606:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:28:45.887018 22152 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:28:45.889510 22413 tablet_service.cc:1467] Processing CreateTablet for tablet dac5fb3568584993a48d099f50fe0b66 (DEFAULT_TABLE table=test-table [id=63d85909dc6246f5b2e3b9b69ec8bd95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:28:45.889590 22274 tablet_service.cc:1467] Processing CreateTablet for tablet dac5fb3568584993a48d099f50fe0b66 (DEFAULT_TABLE table=test-table [id=63d85909dc6246f5b2e3b9b69ec8bd95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:28:45.889708 22413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dac5fb3568584993a48d099f50fe0b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:28:45.889783 22274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dac5fb3568584993a48d099f50fe0b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:28:45.891091 22371 tablet_bootstrap.cc:492] T dac5fb3568584993a48d099f50fe0b66 P fbd16d57fbff4156b4549511ed9319ca: Bootstrap starting.
I20260618 23:28:45.891292 22232 tablet_bootstrap.cc:492] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c: Bootstrap starting.
I20260618 23:28:45.891427 22371 tablet_bootstrap.cc:654] T dac5fb3568584993a48d099f50fe0b66 P fbd16d57fbff4156b4549511ed9319ca: Neither blocks nor log segments found. Creating new log.
I20260618 23:28:45.891577 22232 tablet_bootstrap.cc:654] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c: Neither blocks nor log segments found. Creating new log.
I20260618 23:28:45.891990 22371 tablet_bootstrap.cc:492] T dac5fb3568584993a48d099f50fe0b66 P fbd16d57fbff4156b4549511ed9319ca: No bootstrap required, opened a new log
I20260618 23:28:45.892045 22371 ts_tablet_manager.cc:1397] T dac5fb3568584993a48d099f50fe0b66 P fbd16d57fbff4156b4549511ed9319ca: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:28:45.892140 22232 tablet_bootstrap.cc:492] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c: No bootstrap required, opened a new log
I20260618 23:28:45.892186 22232 ts_tablet_manager.cc:1397] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260618 23:28:45.892473 22371 raft_consensus.cc:348] T dac5fb3568584993a48d099f50fe0b66 P fbd16d57fbff4156b4549511ed9319ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b08fbbb1864186965d51d9738877b8" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 33769 } } peers { permanent_uuid: "fbd16d57fbff4156b4549511ed9319ca" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46529 } } peers { permanent_uuid: "53e617757a0b49e4961839bae4389d7c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45297 } }
I20260618 23:28:45.892534 22232 raft_consensus.cc:348] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b08fbbb1864186965d51d9738877b8" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 33769 } } peers { permanent_uuid: "fbd16d57fbff4156b4549511ed9319ca" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46529 } } peers { permanent_uuid: "53e617757a0b49e4961839bae4389d7c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45297 } }
I20260618 23:28:45.892555 22371 raft_consensus.cc:374] T dac5fb3568584993a48d099f50fe0b66 P fbd16d57fbff4156b4549511ed9319ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:28:45.892601 22371 raft_consensus.cc:729] T dac5fb3568584993a48d099f50fe0b66 P fbd16d57fbff4156b4549511ed9319ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fbd16d57fbff4156b4549511ed9319ca, State: Initialized, Role: FOLLOWER
I20260618 23:28:45.892601 22232 raft_consensus.cc:374] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:28:45.892642 22232 raft_consensus.cc:729] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e617757a0b49e4961839bae4389d7c, State: Initialized, Role: FOLLOWER
I20260618 23:28:45.892650 22371 consensus_queue.cc:260] T dac5fb3568584993a48d099f50fe0b66 P fbd16d57fbff4156b4549511ed9319ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b08fbbb1864186965d51d9738877b8" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 33769 } } peers { permanent_uuid: "fbd16d57fbff4156b4549511ed9319ca" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46529 } } peers { permanent_uuid: "53e617757a0b49e4961839bae4389d7c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45297 } }
I20260618 23:28:45.892694 22232 consensus_queue.cc:260] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b08fbbb1864186965d51d9738877b8" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 33769 } } peers { permanent_uuid: "fbd16d57fbff4156b4549511ed9319ca" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46529 } } peers { permanent_uuid: "53e617757a0b49e4961839bae4389d7c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45297 } }
I20260618 23:28:45.892746 22371 ts_tablet_manager.cc:1428] T dac5fb3568584993a48d099f50fe0b66 P fbd16d57fbff4156b4549511ed9319ca: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260618 23:28:45.892786 22232 ts_tablet_manager.cc:1428] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260618 23:28:45.893771 22347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:45.889285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:33769 (local address 127.0.0.1:59212)
0618 23:28:45.889322 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:45.889333 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:45.889371 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:45.889864 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:45.889865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:45.889867 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:28:45.889945 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:45.889947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.891342 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.891343 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:45.892519 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:45.892522 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.893523 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.893524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:45.893584 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:45.893603 (+ 19us) client_negotiation.cc:769] Sending connection context
0618 23:28:45.893655 (+ 52us) client_negotiation.cc:240] Negotiation successful
0618 23:28:45.893699 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260618 23:28:45.894037 22621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:45.889464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59212 (local address 127.8.39.131:33769)
0618 23:28:45.889667 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:45.889672 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:45.889684 (+ 12us) server_negotiation.cc:407] Connection header received
0618 23:28:45.889723 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:45.889725 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:45.889742 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:45.890031 (+ 289us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:28:45.890096 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.891135 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.892676 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:45.893462 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:45.893526 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:45.893587 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:45.893922 (+ 335us) server_negotiation.cc:299] Negotiation successful
0618 23:28:45.893964 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":111,"threads_started":1}
I20260618 23:28:45.894943 22549 tablet_service.cc:1467] Processing CreateTablet for tablet dac5fb3568584993a48d099f50fe0b66 (DEFAULT_TABLE table=test-table [id=63d85909dc6246f5b2e3b9b69ec8bd95]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:28:45.895310 22549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dac5fb3568584993a48d099f50fe0b66. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:28:45.897998 22624 tablet_bootstrap.cc:492] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8: Bootstrap starting.
I20260618 23:28:45.898599 22624 tablet_bootstrap.cc:654] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8: Neither blocks nor log segments found. Creating new log.
I20260618 23:28:45.899101 22624 log.cc:824] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8: Log is configured to *not* fsync() on all Append() calls
I20260618 23:28:45.899962 22624 tablet_bootstrap.cc:492] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8: No bootstrap required, opened a new log
I20260618 23:28:45.900034 22624 ts_tablet_manager.cc:1397] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:28:45.901501 22624 raft_consensus.cc:348] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b08fbbb1864186965d51d9738877b8" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 33769 } } peers { permanent_uuid: "fbd16d57fbff4156b4549511ed9319ca" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46529 } } peers { permanent_uuid: "53e617757a0b49e4961839bae4389d7c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45297 } }
I20260618 23:28:45.901717 22624 raft_consensus.cc:374] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:28:45.901767 22624 raft_consensus.cc:729] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5b08fbbb1864186965d51d9738877b8, State: Initialized, Role: FOLLOWER
I20260618 23:28:45.901891 22624 consensus_queue.cc:260] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b08fbbb1864186965d51d9738877b8" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 33769 } } peers { permanent_uuid: "fbd16d57fbff4156b4549511ed9319ca" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46529 } } peers { permanent_uuid: "53e617757a0b49e4961839bae4389d7c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45297 } }
I20260618 23:28:45.902190 22624 ts_tablet_manager.cc:1428] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260618 23:28:45.902199 22615 heartbeater.cc:503] Master 127.8.39.190:41745 was elected leader, sending a full tablet report...
W20260618 23:28:45.969408 22480 tablet.cc:2369] T dac5fb3568584993a48d099f50fe0b66 P fbd16d57fbff4156b4549511ed9319ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 23:28:46.065487 22341 tablet.cc:2369] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 23:28:46.114846 22616 tablet.cc:2369] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:28:46.195120 22626 raft_consensus.cc:482] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:28:46.195307 22626 raft_consensus.cc:504] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b08fbbb1864186965d51d9738877b8" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 33769 } } peers { permanent_uuid: "fbd16d57fbff4156b4549511ed9319ca" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46529 } } peers { permanent_uuid: "53e617757a0b49e4961839bae4389d7c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45297 } }
I20260618 23:28:46.195794 22626 leader_election.cc:288] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fbd16d57fbff4156b4549511ed9319ca (127.8.39.130:46529), 53e617757a0b49e4961839bae4389d7c (127.8.39.129:45297)
I20260618 23:28:46.199687 22485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:46.196265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:36829 (local address 127.8.39.130:46529)
0618 23:28:46.196319 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:46.196322 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:46.196443 (+ 121us) server_negotiation.cc:407] Connection header received
0618 23:28:46.196491 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:46.196493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:46.196531 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:46.196616 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:28:46.196917 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:46.197549 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:46.198637 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:46.199167 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:46.199234 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:46.199285 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:46.199535 (+ 250us) server_negotiation.cc:299] Negotiation successful
0618 23:28:46.199571 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260618 23:28:46.199813 22617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:46.196189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:46529 (local address 127.8.39.131:36829)
0618 23:28:46.196302 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:46.196319 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:46.196395 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:46.196676 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:46.196678 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:46.196681 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:28:46.196772 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:46.196774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:46.197660 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:46.197661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:46.198523 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:46.198525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:46.199290 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:46.199292 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:46.199416 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:46.199423 (+ 7us) client_negotiation.cc:769] Sending connection context
0618 23:28:46.199523 (+ 100us) client_negotiation.cc:240] Negotiation successful
0618 23:28:46.199588 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260618 23:28:46.200189 22628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:46.196214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:45297 (local address 127.8.39.131:56583)
0618 23:28:46.196415 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:46.196429 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:46.196523 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:46.196917 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:46.196918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:46.196920 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:28:46.196993 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:46.196995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:46.197781 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:46.197782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:46.198702 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:46.198704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:46.199680 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:46.199681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:46.199777 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:46.199784 (+ 7us) client_negotiation.cc:769] Sending connection context
0618 23:28:46.199843 (+ 59us) client_negotiation.cc:240] Negotiation successful
0618 23:28:46.200022 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":118,"threads_started":1}
I20260618 23:28:46.200475 22433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac5fb3568584993a48d099f50fe0b66" candidate_uuid: "d5b08fbbb1864186965d51d9738877b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbd16d57fbff4156b4549511ed9319ca" is_pre_election: true
I20260618 23:28:46.200562 22627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:46.196243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:56583 (local address 127.8.39.129:45297)
0618 23:28:46.196418 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:46.196421 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:46.196585 (+ 164us) server_negotiation.cc:407] Connection header received
0618 23:28:46.196673 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:46.196674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:46.196707 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:46.196822 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:28:46.197143 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:46.197670 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:46.198869 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:46.199569 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:46.199994 (+ 425us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:46.200378 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:46.200409 (+ 31us) server_negotiation.cc:299] Negotiation successful
0618 23:28:46.200494 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":72,"threads_started":1}
I20260618 23:28:46.200659 22433 raft_consensus.cc:2454] T dac5fb3568584993a48d099f50fe0b66 P fbd16d57fbff4156b4549511ed9319ca [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5b08fbbb1864186965d51d9738877b8 in term 0.
I20260618 23:28:46.200851 22294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac5fb3568584993a48d099f50fe0b66" candidate_uuid: "d5b08fbbb1864186965d51d9738877b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e617757a0b49e4961839bae4389d7c" is_pre_election: true
I20260618 23:28:46.200950 22503 leader_election.cc:302] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d5b08fbbb1864186965d51d9738877b8, fbd16d57fbff4156b4549511ed9319ca; no voters:
I20260618 23:28:46.201007 22294 raft_consensus.cc:2454] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d5b08fbbb1864186965d51d9738877b8 in term 0.
I20260618 23:28:46.201186 22626 raft_consensus.cc:2781] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:28:46.201303 22626 raft_consensus.cc:482] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:28:46.201342 22626 raft_consensus.cc:3037] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:28:46.202149 22626 raft_consensus.cc:504] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b08fbbb1864186965d51d9738877b8" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 33769 } } peers { permanent_uuid: "fbd16d57fbff4156b4549511ed9319ca" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46529 } } peers { permanent_uuid: "53e617757a0b49e4961839bae4389d7c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45297 } }
I20260618 23:28:46.202350 22626 leader_election.cc:288] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8 [CANDIDATE]: Term 1 election: Requested vote from peers fbd16d57fbff4156b4549511ed9319ca (127.8.39.130:46529), 53e617757a0b49e4961839bae4389d7c (127.8.39.129:45297)
I20260618 23:28:46.202541 22433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac5fb3568584993a48d099f50fe0b66" candidate_uuid: "d5b08fbbb1864186965d51d9738877b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fbd16d57fbff4156b4549511ed9319ca"
I20260618 23:28:46.202540 22294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dac5fb3568584993a48d099f50fe0b66" candidate_uuid: "d5b08fbbb1864186965d51d9738877b8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53e617757a0b49e4961839bae4389d7c"
I20260618 23:28:46.202626 22433 raft_consensus.cc:3037] T dac5fb3568584993a48d099f50fe0b66 P fbd16d57fbff4156b4549511ed9319ca [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:28:46.202646 22294 raft_consensus.cc:3037] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:28:46.203336 22294 raft_consensus.cc:2454] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b08fbbb1864186965d51d9738877b8 in term 1.
I20260618 23:28:46.203336 22433 raft_consensus.cc:2454] T dac5fb3568584993a48d099f50fe0b66 P fbd16d57fbff4156b4549511ed9319ca [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5b08fbbb1864186965d51d9738877b8 in term 1.
I20260618 23:28:46.203557 22505 leader_election.cc:302] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 53e617757a0b49e4961839bae4389d7c, d5b08fbbb1864186965d51d9738877b8; no voters:
I20260618 23:28:46.203666 22626 raft_consensus.cc:2781] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:28:46.203817 22626 raft_consensus.cc:686] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8 [term 1 LEADER]: Becoming Leader. State: Replica: d5b08fbbb1864186965d51d9738877b8, State: Running, Role: LEADER
I20260618 23:28:46.203925 22626 consensus_queue.cc:237] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b08fbbb1864186965d51d9738877b8" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 33769 } } peers { permanent_uuid: "fbd16d57fbff4156b4549511ed9319ca" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46529 } } peers { permanent_uuid: "53e617757a0b49e4961839bae4389d7c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45297 } }
I20260618 23:28:46.204772 22152 catalog_manager.cc:5495] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8 reported cstate change: term changed from 0 to 1, leader changed from <none> to d5b08fbbb1864186965d51d9738877b8 (127.8.39.131). New cstate: current_term: 1 leader_uuid: "d5b08fbbb1864186965d51d9738877b8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b08fbbb1864186965d51d9738877b8" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 33769 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fbd16d57fbff4156b4549511ed9319ca" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "53e617757a0b49e4961839bae4389d7c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45297 } health_report { overall_health: UNKNOWN } } }
I20260618 23:28:46.249612 22621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:46.246987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59214 (local address 127.8.39.131:33769)
0618 23:28:46.247063 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:46.247065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:46.247074 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:28:46.247121 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:46.247122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:46.247141 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:46.247228 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:28:46.247368 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:46.247912 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:46.248726 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:46.248944 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:46.249031 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:46.249225 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:28:46.249337 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:28:46.249446 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:46.249466 (+ 20us) server_negotiation.cc:299] Negotiation successful
0618 23:28:46.249487 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260618 23:28:46.251605 22433 raft_consensus.cc:1261] T dac5fb3568584993a48d099f50fe0b66 P fbd16d57fbff4156b4549511ed9319ca [term 1 FOLLOWER]: Refusing update from remote peer d5b08fbbb1864186965d51d9738877b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:28:46.251773 22294 raft_consensus.cc:1261] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [term 1 FOLLOWER]: Refusing update from remote peer d5b08fbbb1864186965d51d9738877b8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:28:46.251937 22626 consensus_queue.cc:1035] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fbd16d57fbff4156b4549511ed9319ca" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:28:46.252122 22626 consensus_queue.cc:1035] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53e617757a0b49e4961839bae4389d7c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:28:46.254817 22634 mvcc.cc:205] Tried to move back new op lower bound from 7298356536322707456 to 7298356536131354624. Current Snapshot: MvccSnapshot[applied={T|T < 7298356536322707456 or (T in {7298356536322707456})}]
I20260618 23:28:46.256220 22151 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:56606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:28:46.256340 22151 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:56606:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:28:46.257076 22151 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca: Sending DeleteTablet for 3 replicas of tablet dac5fb3568584993a48d099f50fe0b66
I20260618 23:28:46.257405 22413 tablet_service.cc:1514] Processing DeleteTablet for tablet dac5fb3568584993a48d099f50fe0b66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:28:46 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:42586
I20260618 23:28:46.257400 22549 tablet_service.cc:1514] Processing DeleteTablet for tablet dac5fb3568584993a48d099f50fe0b66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:28:46 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:59212
I20260618 23:28:46.257609 22641 tablet_replica.cc:330] T dac5fb3568584993a48d099f50fe0b66 P fbd16d57fbff4156b4549511ed9319ca: stopping tablet replica
I20260618 23:28:46.257604 22274 tablet_service.cc:1514] Processing DeleteTablet for tablet dac5fb3568584993a48d099f50fe0b66 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:28:46 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:46226
I20260618 23:28:46.257664 22641 raft_consensus.cc:2229] T dac5fb3568584993a48d099f50fe0b66 P fbd16d57fbff4156b4549511ed9319ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:28:46.257714 22641 raft_consensus.cc:2258] T dac5fb3568584993a48d099f50fe0b66 P fbd16d57fbff4156b4549511ed9319ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:28:46.257925 22643 tablet_replica.cc:330] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c: stopping tablet replica
I20260618 23:28:46.257987 22643 raft_consensus.cc:2229] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:28:46.258029 22643 raft_consensus.cc:2258] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:28:46.258042 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 22190
I20260618 23:28:46.258662 22642 tablet_replica.cc:330] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8: stopping tablet replica
I20260618 23:28:46.258761 22642 raft_consensus.cc:2229] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8 [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:28:46.258870 22642 raft_consensus.cc:2258] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:28:46.259132 22641 ts_tablet_manager.cc:1905] T dac5fb3568584993a48d099f50fe0b66 P fbd16d57fbff4156b4549511ed9319ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:28:46.259320 22642 ts_tablet_manager.cc:1905] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:28:46.260923 22641 ts_tablet_manager.cc:1918] T dac5fb3568584993a48d099f50fe0b66 P fbd16d57fbff4156b4549511ed9319ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:28:46.261061 22641 log.cc:1196] T dac5fb3568584993a48d099f50fe0b66 P fbd16d57fbff4156b4549511ed9319ca: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/wal/wals/dac5fb3568584993a48d099f50fe0b66
I20260618 23:28:46.261322 22642 ts_tablet_manager.cc:1918] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:28:46.261409 22642 log.cc:1196] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/wal/wals/dac5fb3568584993a48d099f50fe0b66
I20260618 23:28:46.261606 22641 ts_tablet_manager.cc:1939] T dac5fb3568584993a48d099f50fe0b66 P fbd16d57fbff4156b4549511ed9319ca: Deleting consensus metadata
I20260618 23:28:46.261773 22642 ts_tablet_manager.cc:1939] T dac5fb3568584993a48d099f50fe0b66 P d5b08fbbb1864186965d51d9738877b8: Deleting consensus metadata
I20260618 23:28:46.262068 22137 catalog_manager.cc:4841] TS fbd16d57fbff4156b4549511ed9319ca (127.8.39.130:46529): tablet dac5fb3568584993a48d099f50fe0b66 (table test-table [id=63d85909dc6246f5b2e3b9b69ec8bd95]) successfully deleted
I20260618 23:28:46.262185 22139 catalog_manager.cc:4841] TS d5b08fbbb1864186965d51d9738877b8 (127.8.39.131:33769): tablet dac5fb3568584993a48d099f50fe0b66 (table test-table [id=63d85909dc6246f5b2e3b9b69ec8bd95]) successfully deleted
W20260618 23:28:46.266448 22139 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.39.129:45297 (error 108)
I20260618 23:28:46.266716 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 22345
I20260618 23:28:46.267108 22347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:46.266966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:45297 (local address 127.0.0.1:46232)
0618 23:28:46.267032 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:46.267068 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.129:45297: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260618 23:28:46.267208 22139 catalog_manager.cc:4584] TS 53e617757a0b49e4961839bae4389d7c (127.8.39.129:45297): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dac5fb3568584993a48d099f50fe0b66: Network error: Client connection negotiation failed: client connection to 127.8.39.129:45297: connect: Connection refused (error 111)
I20260618 23:28:46.271770 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 22483
I20260618 23:28:46.276160 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 22119
I20260618 23:28:46.281687 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:41745
--webserver_interface=127.8.39.190
--webserver_port=46293
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:41745
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:28:46.371984 22645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:28:46.372263 22645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:28:46.372304 22645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:28:46.373556 22645 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:28:46.373641 22645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:28:46.373668 22645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:28:46.373698 22645 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:28:46.373733 22645 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:28:46.375952 22645 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:41745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:41745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=46293
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:28:46.376296 22645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:28:46.376545 22645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:28:46.381759 22651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:28:46.381908 22645 server_base.cc:1034] running on GCE node
W20260618 23:28:46.381759 22653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:46.381788 22650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:28:46.382230 22645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:28:46.382550 22645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:28:46.383762 22645 hybrid_clock.cc:648] HybridClock initialized: now 1781825326383747 us; error 44 us; skew 500 ppm
Jun 18 23:28:46 dist-test-slave-47qt krb5kdc[21573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825326, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:28:46.387029 22645 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:28:46.387555 22645 webserver.cc:460] Webserver started at http://127.8.39.190:46293/ using document root <none> and password file <none>
I20260618 23:28:46.387794 22645 fs_manager.cc:359] Metadata directory not provided
I20260618 23:28:46.387863 22645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:28:46.389477 22645 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:28:46.390336 22660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:46.390707 22645 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:28:46.390938 22645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "b0ea7459b1874d59b31a19576c346dca"
format_stamp: "Formatted at 2026-06-18 23:28:44 on dist-test-slave-47qt"
server_key: "437c6fcb71deec0cbafc9324a92fcb6f"
server_key_iv: "30288963de06b5eae6b057536d2e63f1"
server_key_version: "encryptionkey@0"
I20260618 23:28:46.391461 22645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:28:46.402776 22645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:28:46.405205 22645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:28:46.405364 22645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:28:46.409811 22645 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:41745
I20260618 23:28:46.409845 22712 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:41745 every 8 connection(s)
I20260618 23:28:46.410247 22645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:28:46.411787 22713 sys_catalog.cc:263] Verifying existing consensus state
I20260618 23:28:46.412365 22713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca: Bootstrap starting.
I20260618 23:28:46.416906 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 22645
I20260618 23:28:46.417243 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.129:45297
--local_ip_for_outbound_sockets=127.8.39.129
--tserver_master_addrs=127.8.39.190:41745
--webserver_port=33481
--webserver_interface=127.8.39.129
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:28:46.421816 22713 log.cc:824] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca: Log is configured to *not* fsync() on all Append() calls
I20260618 23:28:46.424052 22713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 23:28:46.424396 22713 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca: Bootstrap complete.
I20260618 23:28:46.426738 22713 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ea7459b1874d59b31a19576c346dca" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41745 } }
I20260618 23:28:46.426961 22713 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0ea7459b1874d59b31a19576c346dca, State: Initialized, Role: FOLLOWER
I20260618 23:28:46.427135 22713 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ea7459b1874d59b31a19576c346dca" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41745 } }
I20260618 23:28:46.427206 22713 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:28:46.427264 22713 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:28:46.427312 22713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [term 2 FOLLOWER]: Advancing to term 3
I20260618 23:28:46.428344 22713 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ea7459b1874d59b31a19576c346dca" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41745 } }
I20260618 23:28:46.428432 22713 leader_election.cc:302] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0ea7459b1874d59b31a19576c346dca; no voters:
I20260618 23:28:46.428653 22713 leader_election.cc:288] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [CANDIDATE]: Term 3 election: Requested vote from peers
I20260618 23:28:46.428731 22717 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [term 3 FOLLOWER]: Leader election won for term 3
I20260618 23:28:46.428867 22717 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [term 3 LEADER]: Becoming Leader. State: Replica: b0ea7459b1874d59b31a19576c346dca, State: Running, Role: LEADER
I20260618 23:28:46.428926 22713 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:28:46.428967 22717 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ea7459b1874d59b31a19576c346dca" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41745 } }
I20260618 23:28:46.429257 22717 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0ea7459b1874d59b31a19576c346dca. Latest consensus state: current_term: 3 leader_uuid: "b0ea7459b1874d59b31a19576c346dca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ea7459b1874d59b31a19576c346dca" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41745 } } }
I20260618 23:28:46.429342 22717 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [sys.catalog]: This master's current role is: LEADER
I20260618 23:28:46.429258 22718 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "b0ea7459b1874d59b31a19576c346dca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0ea7459b1874d59b31a19576c346dca" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41745 } } }
I20260618 23:28:46.429652 22718 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0ea7459b1874d59b31a19576c346dca [sys.catalog]: This master's current role is: LEADER
I20260618 23:28:46.429940 22722 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:28:46.430706 22722 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:28:46.430802 22722 catalog_manager.cc:1259] Loaded cluster ID: 4c8069eb6e0a42d4a48c413ff14e0888
I20260618 23:28:46.430836 22722 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260618 23:28:46.431102 22722 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9418c6a9ad google::LogMessage::Fail() at ??:0
@ 0x7f9418c6d88a google::LogMessage::SendToLog() at ??:0
@ 0x7f9418c6a4de google::LogMessage::Flush() at ??:0
@ 0x7f9418c6ba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9420b8b0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f9420b948b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f941ab380d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f941ab2d954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f941b9916db start_thread at ??:0
@ 0x7f941776861f clone at ??:0
W20260618 23:28:46.515553 22715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:28:46.515779 22715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:28:46.515816 22715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:28:46.517100 22715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:28:46.517244 22715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:28:46.517292 22715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.129
I20260618 23:28:46.518797 22715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.129:45297
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.39.129
--webserver_port=33481
--tserver_master_addrs=127.8.39.190:41745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.129
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:28:46.519069 22715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:28:46.519294 22715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:28:46.525727 22749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:46.525738 22750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:46.525740 22752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:28:46.526110 22715 server_base.cc:1034] running on GCE node
I20260618 23:28:46.526320 22715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:28:46.526552 22715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:28:46.527740 22715 hybrid_clock.cc:648] HybridClock initialized: now 1781825326527726 us; error 29 us; skew 500 ppm
Jun 18 23:28:46 dist-test-slave-47qt krb5kdc[21573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825326, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:28:46.530393 22715 init.cc:377] Logged in from keytab as kudu/127.8.39.129@KRBTEST.COM (short username kudu)
I20260618 23:28:46.530869 22715 webserver.cc:460] Webserver started at http://127.8.39.129:33481/ using document root <none> and password file <none>
I20260618 23:28:46.531203 22715 fs_manager.cc:359] Metadata directory not provided
I20260618 23:28:46.531304 22715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:28:46.532788 22715 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:28:46.533628 22759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:46.533800 22715 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260618 23:28:46.533902 22715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/wal
uuid: "53e617757a0b49e4961839bae4389d7c"
format_stamp: "Formatted at 2026-06-18 23:28:44 on dist-test-slave-47qt"
server_key: "b1003fcb2581ac6d36e977d9574190b9"
server_key_iv: "b7f9506bc75ebb0a1ac891376cbdda1b"
server_key_version: "encryptionkey@0"
I20260618 23:28:46.534401 22715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:28:46.553820 22715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:28:46.556341 22715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:28:46.556499 22715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:28:46.556737 22715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:28:46.557204 22766 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260618 23:28:46.559209 22715 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260618 23:28:46.559291 22715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260618 23:28:46.559340 22715 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260618 23:28:46.559953 22715 ts_tablet_manager.cc:610] Registered 1 tablets
I20260618 23:28:46.560031 22715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260618 23:28:46.560083 22766 tablet_bootstrap.cc:492] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c: Bootstrap starting.
I20260618 23:28:46.567550 22715 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.129:45297
I20260618 23:28:46.567658 22873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.129:45297 every 8 connection(s)
I20260618 23:28:46.568001 22715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
I20260618 23:28:46.571784 22766 log.cc:824] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c: Log is configured to *not* fsync() on all Append() calls
I20260618 23:28:46.573287 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 22715
I20260618 23:28:46.573549 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.130:46529
--local_ip_for_outbound_sockets=127.8.39.130
--tserver_master_addrs=127.8.39.190:41745
--webserver_port=43549
--webserver_interface=127.8.39.130
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:28:46.574741 22766 tablet_bootstrap.cc:492] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 23:28:46.575075 22766 tablet_bootstrap.cc:492] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c: Bootstrap complete.
I20260618 23:28:46.576331 22766 ts_tablet_manager.cc:1397] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.012s
I20260618 23:28:46.577864 22766 raft_consensus.cc:348] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b08fbbb1864186965d51d9738877b8" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 33769 } } peers { permanent_uuid: "fbd16d57fbff4156b4549511ed9319ca" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46529 } } peers { permanent_uuid: "53e617757a0b49e4961839bae4389d7c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45297 } }
I20260618 23:28:46.578258 22766 raft_consensus.cc:729] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53e617757a0b49e4961839bae4389d7c, State: Initialized, Role: FOLLOWER
I20260618 23:28:46.578429 22766 consensus_queue.cc:260] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b08fbbb1864186965d51d9738877b8" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 33769 } } peers { permanent_uuid: "fbd16d57fbff4156b4549511ed9319ca" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46529 } } peers { permanent_uuid: "53e617757a0b49e4961839bae4389d7c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45297 } }
I20260618 23:28:46.578683 22766 ts_tablet_manager.cc:1428] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c: Time spent starting tablet: real 0.002s user 0.003s sys 0.002s
W20260618 23:28:46.661224 22876 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:28:46.568743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.129:53227)
0618 23:28:46.568937 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:46.568973 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:46.569513 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:46.660954 (+ 91441us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":103,"threads_started":1}
W20260618 23:28:46.661450 22874 heartbeater.cc:644] Failed to heartbeat to 127.8.39.190:41745 (0 consecutive failures): Network error: Failed to ping master at 127.8.39.190:41745: Client connection negotiation failed: client connection to 127.8.39.190:41745: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260618 23:28:46.661777 22876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:46.661659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.129:37519)
0618 23:28:46.661714 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:46.661750 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260618 23:28:46.662143 22876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:46.662059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.129:49045)
0618 23:28:46.662107 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:46.662129 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260618 23:28:46.662271 22874 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260618 23:28:46.667701 22878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:28:46.667891 22878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:28:46.667914 22878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:28:46.669292 22878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:28:46.669338 22878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:28:46.669366 22878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.130
I20260618 23:28:46.670744 22878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.130:46529
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.39.130
--webserver_port=43549
--tserver_master_addrs=127.8.39.190:41745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.130
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:28:46.670924 22878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:28:46.671105 22878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:28:46.676836 22887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:46.676849 22884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:46.676869 22885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:28:46.677073 22878 server_base.cc:1034] running on GCE node
I20260618 23:28:46.677380 22878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:28:46.677604 22878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:28:46.678743 22878 hybrid_clock.cc:648] HybridClock initialized: now 1781825326678714 us; error 39 us; skew 500 ppm
Jun 18 23:28:46 dist-test-slave-47qt krb5kdc[21573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825326, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:28:46.681288 22878 init.cc:377] Logged in from keytab as kudu/127.8.39.130@KRBTEST.COM (short username kudu)
I20260618 23:28:46.681740 22878 webserver.cc:460] Webserver started at http://127.8.39.130:43549/ using document root <none> and password file <none>
I20260618 23:28:46.681962 22878 fs_manager.cc:359] Metadata directory not provided
I20260618 23:28:46.682005 22878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:28:46.683305 22878 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:28:46.683902 22894 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:46.684191 22878 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:28:46.684278 22878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/wal
uuid: "fbd16d57fbff4156b4549511ed9319ca"
format_stamp: "Formatted at 2026-06-18 23:28:44 on dist-test-slave-47qt"
server_key: "3fe41cd30ff79803d61fc86ca00f8291"
server_key_iv: "15032fbbeac4574ce0161e3b2f178cd7"
server_key_version: "encryptionkey@0"
I20260618 23:28:46.684639 22878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:28:46.702631 22878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:28:46.704897 22878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:28:46.705013 22878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:28:46.705439 22878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:28:46.705758 22878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:28:46.705811 22878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:46.705845 22878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:28:46.705873 22878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:46.712230 22878 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.130:46529
I20260618 23:28:46.712270 23007 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.130:46529 every 8 connection(s)
I20260618 23:28:46.712625 22878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
I20260618 23:28:46.713699 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:46.713333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.130:56941)
0618 23:28:46.713487 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:46.713551 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":74,"threads_started":1}
W20260618 23:28:46.714088 23008 heartbeater.cc:644] Failed to heartbeat to 127.8.39.190:41745 (0 consecutive failures): Network error: Failed to ping master at 127.8.39.190:41745: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
I20260618 23:28:46.714382 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:46.714295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.130:43775)
0618 23:28:46.714342 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:46.714366 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260618 23:28:46.714645 23010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:46.714586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.130:51519)
0618 23:28:46.714606 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:46.714634 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260618 23:28:46.714771 23008 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260618 23:28:46.719151 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 22878
I20260618 23:28:46.719405 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.131:33769
--local_ip_for_outbound_sockets=127.8.39.131
--tserver_master_addrs=127.8.39.190:41745
--webserver_port=40087
--webserver_interface=127.8.39.131
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:28:46.804785 23011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:28:46.805011 23011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:28:46.805037 23011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:28:46.806589 23011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:28:46.806650 23011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:28:46.806682 23011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.131
I20260618 23:28:46.808346 23011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.131:33769
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.39.131
--webserver_port=40087
--tserver_master_addrs=127.8.39.190:41745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.131
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:28:46.808599 23011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:28:46.808884 23011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:28:46.813980 23017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:28:46.813982 23016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:28:46.814148 23011 server_base.cc:1034] running on GCE node
W20260618 23:28:46.814189 23019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:28:46.814426 23011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:28:46.814677 23011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:28:46.815850 23011 hybrid_clock.cc:648] HybridClock initialized: now 1781825326815831 us; error 30 us; skew 500 ppm
Jun 18 23:28:46 dist-test-slave-47qt krb5kdc[21573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825326, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:28:46.818815 23011 init.cc:377] Logged in from keytab as kudu/127.8.39.131@KRBTEST.COM (short username kudu)
I20260618 23:28:46.819268 23011 webserver.cc:460] Webserver started at http://127.8.39.131:40087/ using document root <none> and password file <none>
I20260618 23:28:46.819603 23011 fs_manager.cc:359] Metadata directory not provided
I20260618 23:28:46.819684 23011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:28:46.821085 23011 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:28:46.822120 23026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:46.822489 23011 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:28:46.822621 23011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/wal
uuid: "d5b08fbbb1864186965d51d9738877b8"
format_stamp: "Formatted at 2026-06-18 23:28:45 on dist-test-slave-47qt"
server_key: "4d16de477a96d86f015d8a1824dbe10f"
server_key_iv: "5936a638752d8e0b2db368433c376bff"
server_key_version: "encryptionkey@0"
I20260618 23:28:46.823045 23011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:28:46.842778 23011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:28:46.845615 23011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:28:46.845810 23011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:28:46.846153 23011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:28:46.846529 23011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:28:46.846589 23011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:46.846623 23011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:28:46.846647 23011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:28:46.852752 23011 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.131:33769
I20260618 23:28:46.852892 23139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.131:33769 every 8 connection(s)
I20260618 23:28:46.853186 23011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
I20260618 23:28:46.854240 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:46.853836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.131:43389)
0618 23:28:46.854001 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:46.854080 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":74,"threads_started":1}
I20260618 23:28:46.854620 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 23011
W20260618 23:28:46.854667 23140 heartbeater.cc:644] Failed to heartbeat to 127.8.39.190:41745 (0 consecutive failures): Network error: Failed to ping master at 127.8.39.190:41745: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
I20260618 23:28:46.855000 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:46.854866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.131:33721)
0618 23:28:46.854936 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:46.854968 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260618 23:28:46.855296 23142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:46.855222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.131:55353)
0618 23:28:46.855250 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:46.855274 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260618 23:28:46.855401 23140 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260618 23:28:47.663002 23144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:47.662701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.129:35555)
0618 23:28:47.662869 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:47.662940 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":77,"threads_started":1}
I20260618 23:28:47.715369 23145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:47.715125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.130:44533)
0618 23:28:47.715263 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:47.715303 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":59,"threads_started":1}
I20260618 23:28:47.856031 23146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:47.855748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.131:45561)
0618 23:28:47.855925 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:47.855981 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":86,"threads_started":1}
I20260618 23:28:48.272503 23147 raft_consensus.cc:482] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:28:48.272619 23147 raft_consensus.cc:504] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b08fbbb1864186965d51d9738877b8" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 33769 } } peers { permanent_uuid: "fbd16d57fbff4156b4549511ed9319ca" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46529 } } peers { permanent_uuid: "53e617757a0b49e4961839bae4389d7c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45297 } }
I20260618 23:28:48.272989 23147 leader_election.cc:288] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d5b08fbbb1864186965d51d9738877b8 (127.8.39.131:33769), fbd16d57fbff4156b4549511ed9319ca (127.8.39.130:46529)
Jun 18 23:28:48 dist-test-slave-47qt krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825326, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for kudu/127.8.39.131@KRBTEST.COM
Jun 18 23:28:48 dist-test-slave-47qt krb5kdc[21573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825326, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for kudu/127.8.39.130@KRBTEST.COM
I20260618 23:28:48.286558 23149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:48.273166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:46529 (local address 127.8.39.129:44915)
0618 23:28:48.273457 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:48.273467 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:48.273523 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:48.274437 (+ 914us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:48.274440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:48.274955 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:28:48.275230 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:48.275237 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:48.276916 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:48.276917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:48.278240 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:48.278243 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:48.279110 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:48.279110 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:48.279253 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:48.279836 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:28:48.279964 (+ 128us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:28:48.282802 (+ 2838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:28:48.284568 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:48.284576 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:48.284579 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:48.284917 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:48.285200 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:48.285201 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:48.285201 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:48.285330 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:48.285795 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:28:48.285812 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:28:48.286018 (+ 206us) client_negotiation.cc:769] Sending connection context
0618 23:28:48.286147 (+ 129us) client_negotiation.cc:240] Negotiation successful
0618 23:28:48.286453 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":110,"thread_start_us":59,"threads_started":1}
I20260618 23:28:48.286774 23151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:48.273462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:46997 (local address 127.8.39.131:33769)
0618 23:28:48.273633 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:48.273640 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:48.273669 (+ 29us) server_negotiation.cc:407] Connection header received
0618 23:28:48.274197 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:48.274199 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:48.274237 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:48.274358 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:48.275389 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:48.276782 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:48.278359 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:48.279001 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:48.279109 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:48.282385 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:48.282390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:48.282390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:48.282402 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:48.284399 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:48.285014 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:48.285017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:48.285017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:48.285108 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:48.285410 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:48.285411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:48.285411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:48.286019 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:48.286256 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:48.286481 (+ 225us) server_negotiation.cc:299] Negotiation successful
0618 23:28:48.286647 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":89,"threads_started":1}
I20260618 23:28:48.286880 23150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:48.273301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:44915 (local address 127.8.39.130:46529)
0618 23:28:48.273442 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0618 23:28:48.273452 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 23:28:48.273514 (+ 62us) server_negotiation.cc:407] Connection header received
0618 23:28:48.274143 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:28:48.274147 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:28:48.274203 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:28:48.274358 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:28:48.275389 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:48.276785 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:48.278359 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:28:48.278989 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:28:48.279110 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:48.282923 (+ 3813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:28:48.282927 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:28:48.282927 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:28:48.282936 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:28:48.284441 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:48.285013 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:48.285023 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:48.285023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:48.285103 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:28:48.285406 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:28:48.285406 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:28:48.285407 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:28:48.285651 (+ 244us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:28:48.285715 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0618 23:28:48.286294 (+ 579us) server_negotiation.cc:299] Negotiation successful
0618 23:28:48.286602 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":60,"threads_started":1}
I20260618 23:28:48.287478 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:48.273166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:33769 (local address 127.8.39.129:46997)
0618 23:28:48.273273 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:48.273283 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:28:48.273327 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:28:48.274437 (+ 1110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:28:48.274440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:28:48.274957 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:28:48.275230 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:48.275237 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:48.276914 (+ 1677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:48.276915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:48.278240 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:28:48.278243 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:28:48.279109 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:28:48.279110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:28:48.279253 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:28:48.280011 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:28:48.280019 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:28:48.282223 (+ 2204us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:28:48.284619 (+ 2396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:48.284620 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:48.284620 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:48.284916 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:48.285205 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:28:48.285206 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:28:48.285206 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:28:48.285330 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:28:48.286207 (+ 877us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:28:48.286208 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:28:48.286270 (+ 62us) client_negotiation.cc:769] Sending connection context
0618 23:28:48.286375 (+ 105us) client_negotiation.cc:240] Negotiation successful
0618 23:28:48.286597 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":74,"thread_start_us":52,"threads_started":1}
W20260618 23:28:48.287757 22761 leader_election.cc:341] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fbd16d57fbff4156b4549511ed9319ca (127.8.39.130:46529): Not found: Tablet not found: dac5fb3568584993a48d099f50fe0b66
W20260618 23:28:48.288161 22763 leader_election.cc:341] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d5b08fbbb1864186965d51d9738877b8 (127.8.39.131:33769): Not found: Tablet not found: dac5fb3568584993a48d099f50fe0b66
I20260618 23:28:48.288234 22763 leader_election.cc:302] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53e617757a0b49e4961839bae4389d7c; no voters: d5b08fbbb1864186965d51d9738877b8, fbd16d57fbff4156b4549511ed9319ca
I20260618 23:28:48.288335 23147 raft_consensus.cc:2726] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 23:28:48.664011 23148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:48.663663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.129:56069)
0618 23:28:48.663890 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:48.663955 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198}
I20260618 23:28:48.716212 23152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:48.715958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.130:36353)
0618 23:28:48.716107 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:48.716165 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":61,"threads_started":1}
I20260618 23:28:48.856778 23153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:48.856533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.131:57387)
0618 23:28:48.856702 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:48.856738 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":90,"threads_started":1}
I20260618 23:28:49.665030 23154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:49.664665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.129:57821)
0618 23:28:49.664949 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:49.664984 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":62,"threads_started":1}
I20260618 23:28:49.717067 23155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:49.716844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.130:39817)
0618 23:28:49.716978 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:49.717021 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":59,"threads_started":1}
I20260618 23:28:49.857511 23156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:49.857302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.131:39665)
0618 23:28:49.857437 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:49.857472 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":60,"threads_started":1}
I20260618 23:28:50.329977 23157 raft_consensus.cc:482] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:28:50.330070 23157 raft_consensus.cc:504] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b08fbbb1864186965d51d9738877b8" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 33769 } } peers { permanent_uuid: "fbd16d57fbff4156b4549511ed9319ca" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46529 } } peers { permanent_uuid: "53e617757a0b49e4961839bae4389d7c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45297 } }
I20260618 23:28:50.330288 23157 leader_election.cc:288] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d5b08fbbb1864186965d51d9738877b8 (127.8.39.131:33769), fbd16d57fbff4156b4549511ed9319ca (127.8.39.130:46529)
W20260618 23:28:50.330739 22761 leader_election.cc:341] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fbd16d57fbff4156b4549511ed9319ca (127.8.39.130:46529): Not found: Tablet not found: dac5fb3568584993a48d099f50fe0b66
W20260618 23:28:50.330817 22763 leader_election.cc:341] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d5b08fbbb1864186965d51d9738877b8 (127.8.39.131:33769): Not found: Tablet not found: dac5fb3568584993a48d099f50fe0b66
I20260618 23:28:50.330924 22763 leader_election.cc:302] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53e617757a0b49e4961839bae4389d7c; no voters: d5b08fbbb1864186965d51d9738877b8, fbd16d57fbff4156b4549511ed9319ca
I20260618 23:28:50.331012 23157 raft_consensus.cc:2726] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 23:28:50.665963 23158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:50.665566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.129:60889)
0618 23:28:50.665799 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:50.665873 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":69,"threads_started":1}
I20260618 23:28:50.717738 23159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:50.717534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.130:46893)
0618 23:28:50.717651 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:50.717686 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":59,"threads_started":1}
I20260618 23:28:50.858242 23160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:50.858017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.131:42529)
0618 23:28:50.858159 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:50.858201 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":63,"threads_started":1}
I20260618 23:28:51.666944 23161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:51.666629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.129:34565)
0618 23:28:51.666812 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:51.666878 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":62,"threads_started":1}
I20260618 23:28:51.718384 23162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:51.718161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.130:51871)
0618 23:28:51.718311 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:51.718344 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":70,"threads_started":1}
I20260618 23:28:51.859063 23163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:51.858846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.131:38503)
0618 23:28:51.858986 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:51.859023 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
I20260618 23:28:52.667999 23164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:52.667712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.129:45185)
0618 23:28:52.667863 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:52.667907 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":76,"threads_started":1}
I20260618 23:28:52.719113 23165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:52.718872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.130:37233)
0618 23:28:52.719027 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:52.719068 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":64,"threads_started":1}
I20260618 23:28:52.860014 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:52.859786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.131:48499)
0618 23:28:52.859925 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:52.859969 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":62,"threads_started":1}
I20260618 23:28:52.954757 23167 raft_consensus.cc:482] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:28:52.954895 23167 raft_consensus.cc:504] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b08fbbb1864186965d51d9738877b8" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 33769 } } peers { permanent_uuid: "fbd16d57fbff4156b4549511ed9319ca" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46529 } } peers { permanent_uuid: "53e617757a0b49e4961839bae4389d7c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45297 } }
I20260618 23:28:52.955096 23167 leader_election.cc:288] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d5b08fbbb1864186965d51d9738877b8 (127.8.39.131:33769), fbd16d57fbff4156b4549511ed9319ca (127.8.39.130:46529)
W20260618 23:28:52.955519 22761 leader_election.cc:341] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fbd16d57fbff4156b4549511ed9319ca (127.8.39.130:46529): Not found: Tablet not found: dac5fb3568584993a48d099f50fe0b66
W20260618 23:28:52.955605 22763 leader_election.cc:341] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d5b08fbbb1864186965d51d9738877b8 (127.8.39.131:33769): Not found: Tablet not found: dac5fb3568584993a48d099f50fe0b66
I20260618 23:28:52.955720 22763 leader_election.cc:302] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53e617757a0b49e4961839bae4389d7c; no voters: d5b08fbbb1864186965d51d9738877b8, fbd16d57fbff4156b4549511ed9319ca
I20260618 23:28:52.955813 23167 raft_consensus.cc:2726] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 23:28:53.668848 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:53.668567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.129:51365)
0618 23:28:53.668738 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:53.668799 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":75,"threads_started":1}
I20260618 23:28:53.719904 23169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:53.719650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.130:54793)
0618 23:28:53.719818 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:53.719858 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":60,"threads_started":1}
I20260618 23:28:53.860976 23170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:53.860599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.131:59821)
0618 23:28:53.860735 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:53.860786 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":85,"threads_started":1}
I20260618 23:28:54.670059 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:54.669646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.129:47913)
0618 23:28:54.669975 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:54.670013 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":67,"threads_started":1}
I20260618 23:28:54.720731 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:54.720478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.130:41057)
0618 23:28:54.720642 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:54.720683 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":57,"threads_started":1}
I20260618 23:28:54.861773 23173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:54.861548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.131:35427)
0618 23:28:54.861692 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:54.861732 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":85,"threads_started":1}
I20260618 23:28:55.670908 23174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:55.670684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.129:40441)
0618 23:28:55.670826 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:55.670862 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
I20260618 23:28:55.721715 23175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:55.721416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.130:47461)
0618 23:28:55.721593 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:55.721633 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":78,"threads_started":1}
I20260618 23:28:55.862470 23176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:55.862253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.131:48001)
0618 23:28:55.862382 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:55.862423 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":56,"threads_started":1}
I20260618 23:28:56.470245 23177 raft_consensus.cc:482] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:28:56.470391 23177 raft_consensus.cc:504] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5b08fbbb1864186965d51d9738877b8" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 33769 } } peers { permanent_uuid: "fbd16d57fbff4156b4549511ed9319ca" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46529 } } peers { permanent_uuid: "53e617757a0b49e4961839bae4389d7c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 45297 } }
I20260618 23:28:56.470803 23177 leader_election.cc:288] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d5b08fbbb1864186965d51d9738877b8 (127.8.39.131:33769), fbd16d57fbff4156b4549511ed9319ca (127.8.39.130:46529)
W20260618 23:28:56.471235 22763 leader_election.cc:341] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d5b08fbbb1864186965d51d9738877b8 (127.8.39.131:33769): Not found: Tablet not found: dac5fb3568584993a48d099f50fe0b66
W20260618 23:28:56.471314 22761 leader_election.cc:341] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer fbd16d57fbff4156b4549511ed9319ca (127.8.39.130:46529): Not found: Tablet not found: dac5fb3568584993a48d099f50fe0b66
I20260618 23:28:56.471357 22761 leader_election.cc:302] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 53e617757a0b49e4961839bae4389d7c; no voters: d5b08fbbb1864186965d51d9738877b8, fbd16d57fbff4156b4549511ed9319ca
I20260618 23:28:56.471581 23177 raft_consensus.cc:2726] T dac5fb3568584993a48d099f50fe0b66 P 53e617757a0b49e4961839bae4389d7c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260618 23:28:56.671655 23178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:56.671422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.129:47293)
0618 23:28:56.671564 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:56.671604 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":61,"threads_started":1}
I20260618 23:28:56.722400 23179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:56.722175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.130:44585)
0618 23:28:56.722317 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:56.722355 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":59,"threads_started":1}
I20260618 23:28:56.863234 23180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:56.863008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.131:49071)
0618 23:28:56.863151 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:56.863189 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
I20260618 23:28:57.672452 23181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:57.672203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.129:39381)
0618 23:28:57.672348 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:57.672402 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":61,"threads_started":1}
I20260618 23:28:57.723153 23182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:57.722912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.130:55067)
0618 23:28:57.723059 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:57.723107 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":58,"threads_started":1}
I20260618 23:28:57.864176 23183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:57.863907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.131:55517)
0618 23:28:57.864047 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:57.864086 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260618 23:28:58.673323 23184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:58.672993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.129:34077)
0618 23:28:58.673225 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:58.673264 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":84,"threads_started":1}
I20260618 23:28:58.724097 23185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:58.723801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.130:50355)
0618 23:28:58.724003 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:58.724048 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":85,"threads_started":1}
I20260618 23:28:58.865011 23186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:58.864786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.131:52943)
0618 23:28:58.864927 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:58.864967 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":63,"threads_started":1}
I20260618 23:28:59.674170 23187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:59.673932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.129:56961)
0618 23:28:59.674074 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:59.674120 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260618 23:28:59.724898 23188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:59.724699 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.130:49687)
0618 23:28:59.724815 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:59.724852 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":56,"threads_started":1}
I20260618 23:28:59.865887 23189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:28:59.865657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.131:42487)
0618 23:28:59.865810 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0618 23:28:59.865843 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":76,"threads_started":1}
I20260618 23:29:00.674865 23190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:00.674645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.129:45187)
0618 23:29:00.674783 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:00.674818 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":61,"threads_started":1}
I20260618 23:29:00.725656 23191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:00.725413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.130:42321)
0618 23:29:00.725567 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:00.725609 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":63,"threads_started":1}
I20260618 23:29:00.866675 23192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:00.866448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.131:44249)
0618 23:29:00.866592 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:00.866629 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":63,"threads_started":1}
I20260618 23:29:01.675707 23193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:01.675463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.129:53561)
0618 23:29:01.675625 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:01.675654 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":87,"threads_started":1}
I20260618 23:29:01.726419 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:01.726193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41745 (local address 127.8.39.130:58307)
0618 23:29:01.726343 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:01.726376 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:41745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
I20260618 23:29:01.856386 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 22715
I20260618 23:29:01.861380 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 22878
I20260618 23:29:01.866395 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 23011
2026-06-18T23:29:01Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18641 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:29:01 dist-test-slave-47qt krb5kdc[23197](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:29:01 dist-test-slave-47qt krb5kdc[23197](info): set up 2 sockets
Jun 18 23:29:01 dist-test-slave-47qt krb5kdc[23197](info): commencing operation
krb5kdc: starting...
W20260618 23:29:02.933511 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:29:03 dist-test-slave-47qt krb5kdc[23197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:29:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:29:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.39.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
I20260618 23:29:03.064894 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:37277
--webserver_interface=127.8.39.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:37277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:29:03.151719 23210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:29:03.151928 23210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:29:03.151954 23210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:29:03.153221 23210 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260618 23:29:03.153264 23210 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260618 23:29:03.153277 23210 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260618 23:29:03.153292 23210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:29:03.153306 23210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:29:03.153347 23210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:29:03.153378 23210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:29:03.153394 23210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:29:03.154906 23210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:37277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:37277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:29:03.155179 23210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:29:03.155416 23210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:29:03.160817 23218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:29:03.160917 23215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:29:03.160919 23210 server_base.cc:1034] running on GCE node
W20260618 23:29:03.160843 23216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:29:03.161381 23210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:29:03.161707 23210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:29:03.162881 23210 hybrid_clock.cc:648] HybridClock initialized: now 1781825343162866 us; error 44 us; skew 500 ppm
Jun 18 23:29:03 dist-test-slave-47qt krb5kdc[23197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825343, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:29:03.165431 23210 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:29:03.166214 23210 webserver.cc:460] Webserver started at http://127.8.39.190:40913/ using document root <none> and password file <none>
I20260618 23:29:03.166531 23210 fs_manager.cc:359] Metadata directory not provided
I20260618 23:29:03.166587 23210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:29:03.166702 23210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:29:03.168053 23210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "28d09a3f812648c487d8fb8164065f7f"
format_stamp: "Formatted at 2026-06-18 23:29:03 on dist-test-slave-47qt"
server_key: "06db7f3f59455f9cbcd91986c295560a"
server_key_iv: "39715f061ebe9dc819ce3b157cfd1ba9"
server_key_version: "encryptionkey@0"
I20260618 23:29:03.168413 23210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "28d09a3f812648c487d8fb8164065f7f"
format_stamp: "Formatted at 2026-06-18 23:29:03 on dist-test-slave-47qt"
server_key: "06db7f3f59455f9cbcd91986c295560a"
server_key_iv: "39715f061ebe9dc819ce3b157cfd1ba9"
server_key_version: "encryptionkey@0"
I20260618 23:29:03.169981 23210 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:29:03.170670 23225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:29:03.171100 23210 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:29:03.171216 23210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "28d09a3f812648c487d8fb8164065f7f"
format_stamp: "Formatted at 2026-06-18 23:29:03 on dist-test-slave-47qt"
server_key: "06db7f3f59455f9cbcd91986c295560a"
server_key_iv: "39715f061ebe9dc819ce3b157cfd1ba9"
server_key_version: "encryptionkey@0"
I20260618 23:29:03.171298 23210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:29:03.185287 23210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:29:03.188131 23210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:29:03.188287 23210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:29:03.193483 23210 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:37277
I20260618 23:29:03.193692 23277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:37277 every 8 connection(s)
I20260618 23:29:03.193933 23210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:29:03.194818 23278 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:29:03.197317 23278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f: Bootstrap starting.
I20260618 23:29:03.197925 23278 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f: Neither blocks nor log segments found. Creating new log.
I20260618 23:29:03.198240 23278 log.cc:824] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f: Log is configured to *not* fsync() on all Append() calls
I20260618 23:29:03.198927 23278 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f: No bootstrap required, opened a new log
I20260618 23:29:03.200320 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 23210
I20260618 23:29:03.200456 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:29:03.200646 8350 external_mini_cluster.cc:1428] Setting key 2cf15515736f75b696f333ace8bf7c20
I20260618 23:29:03.200695 23278 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d09a3f812648c487d8fb8164065f7f" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 37277 } }
I20260618 23:29:03.200891 23278 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:29:03.200937 23278 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28d09a3f812648c487d8fb8164065f7f, State: Initialized, Role: FOLLOWER
I20260618 23:29:03.201083 23278 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d09a3f812648c487d8fb8164065f7f" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 37277 } }
I20260618 23:29:03.201191 23278 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:29:03.201231 23278 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:29:03.201282 23278 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:29:03.202139 23278 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d09a3f812648c487d8fb8164065f7f" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 37277 } }
I20260618 23:29:03.202265 23278 leader_election.cc:302] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28d09a3f812648c487d8fb8164065f7f; no voters:
I20260618 23:29:03.202450 23278 leader_election.cc:288] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:29:03.202477 23283 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:29:03.202584 23283 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [term 1 LEADER]: Becoming Leader. State: Replica: 28d09a3f812648c487d8fb8164065f7f, State: Running, Role: LEADER
I20260618 23:29:03.202719 23278 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:29:03.202666 23283 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d09a3f812648c487d8fb8164065f7f" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 37277 } }
I20260618 23:29:03.203034 23283 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28d09a3f812648c487d8fb8164065f7f. Latest consensus state: current_term: 1 leader_uuid: "28d09a3f812648c487d8fb8164065f7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d09a3f812648c487d8fb8164065f7f" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 37277 } } }
I20260618 23:29:03.203118 23283 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [sys.catalog]: This master's current role is: LEADER
I20260618 23:29:03.203044 23284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28d09a3f812648c487d8fb8164065f7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d09a3f812648c487d8fb8164065f7f" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 37277 } } }
I20260618 23:29:03.203328 23284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [sys.catalog]: This master's current role is: LEADER
I20260618 23:29:03.203465 23288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:29:03.204262 23288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:29:03.206176 23288 catalog_manager.cc:1347] Generated new cluster ID: 9cc7d5021b2f4aed9925d525df5fda1c
I20260618 23:29:03.206251 23288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 18 23:29:03 dist-test-slave-47qt krb5kdc[23197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825343, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:29:03.213517 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:03.201172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59188 (local address 127.8.39.190:37277)
0618 23:29:03.201362 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0618 23:29:03.201371 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 23:29:03.201402 (+ 31us) server_negotiation.cc:407] Connection header received
0618 23:29:03.201511 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:29:03.201517 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:29:03.201558 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:29:03.201685 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:29:03.202292 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.203694 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.204886 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.205533 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.205642 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:03.208866 (+ 3224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:29:03.208871 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:29:03.208871 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:29:03.208883 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:29:03.211452 (+ 2569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:29:03.211897 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:29:03.211900 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:29:03.211900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:29:03.212026 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:29:03.212241 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:29:03.212242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:29:03.212242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:29:03.212553 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:29:03.213089 (+ 536us) server_negotiation.cc:1035] Waiting for connection context
0618 23:29:03.213178 (+ 89us) server_negotiation.cc:299] Negotiation successful
0618 23:29:03.213312 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":89,"threads_started":1}
I20260618 23:29:03.220140 23288 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:29:03.220669 23288 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:29:03.230077 23288 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f: Generated new TSK 0
I20260618 23:29:03.230268 23288 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.39.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
I20260618 23:29:03.264636 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.129:0
--local_ip_for_outbound_sockets=127.8.39.129
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:37277
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:29:03.364751 23304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:29:03.364965 23304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:29:03.364992 23304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:29:03.366240 23304 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:29:03.366298 23304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:29:03.366330 23304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.129
I20260618 23:29:03.367928 23304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:37277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23304
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.129
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:29:03.368191 23304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:29:03.368474 23304 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:29:03.374277 23310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:29:03.374305 23312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:29:03.374297 23309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:29:03.374475 23304 server_base.cc:1034] running on GCE node
I20260618 23:29:03.374766 23304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:29:03.375039 23304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:29:03.376284 23304 hybrid_clock.cc:648] HybridClock initialized: now 1781825343376258 us; error 26 us; skew 500 ppm
Jun 18 23:29:03 dist-test-slave-47qt krb5kdc[23197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825343, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:29:03.379429 23304 init.cc:377] Logged in from keytab as kudu/127.8.39.129@KRBTEST.COM (short username kudu)
I20260618 23:29:03.379930 23304 webserver.cc:460] Webserver started at http://127.8.39.129:44159/ using document root <none> and password file <none>
I20260618 23:29:03.380179 23304 fs_manager.cc:359] Metadata directory not provided
I20260618 23:29:03.380224 23304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:29:03.380391 23304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:29:03.381561 23304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/data/instance:
uuid: "f0eff757ff2c48608da677b47879e369"
format_stamp: "Formatted at 2026-06-18 23:29:03 on dist-test-slave-47qt"
server_key: "1acb970c37529997eee9ea3ba9447865"
server_key_iv: "aaf8994287dcaa88f275596a9d1ac3e0"
server_key_version: "encryptionkey@0"
I20260618 23:29:03.381918 23304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance:
uuid: "f0eff757ff2c48608da677b47879e369"
format_stamp: "Formatted at 2026-06-18 23:29:03 on dist-test-slave-47qt"
server_key: "1acb970c37529997eee9ea3ba9447865"
server_key_iv: "aaf8994287dcaa88f275596a9d1ac3e0"
server_key_version: "encryptionkey@0"
I20260618 23:29:03.383265 23304 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:29:03.384093 23319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:29:03.384342 23304 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:29:03.384409 23304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/wal
uuid: "f0eff757ff2c48608da677b47879e369"
format_stamp: "Formatted at 2026-06-18 23:29:03 on dist-test-slave-47qt"
server_key: "1acb970c37529997eee9ea3ba9447865"
server_key_iv: "aaf8994287dcaa88f275596a9d1ac3e0"
server_key_version: "encryptionkey@0"
I20260618 23:29:03.384529 23304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:29:03.405874 23304 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:29:03.408150 23304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:29:03.408311 23304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:29:03.408644 23304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:29:03.409003 23304 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:29:03.409065 23304 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:29:03.409098 23304 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:29:03.409119 23304 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:29:03.415901 23304 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.129:44949
I20260618 23:29:03.416003 23432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.129:44949 every 8 connection(s)
I20260618 23:29:03.416342 23304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
I20260618 23:29:03.420606 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 23304
I20260618 23:29:03.420709 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance
I20260618 23:29:03.420900 8350 external_mini_cluster.cc:1428] Setting key 30e1bd261d78b3bdc4c3c011836e524f
Jun 18 23:29:03 dist-test-slave-47qt krb5kdc[23197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825343, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:29:03.428395 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:03.417427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:49177 (local address 127.8.39.190:37277)
0618 23:29:03.417521 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0618 23:29:03.417523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:29:03.417818 (+ 295us) server_negotiation.cc:407] Connection header received
0618 23:29:03.418390 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:29:03.418392 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:29:03.418405 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:29:03.418455 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:29:03.419434 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.419950 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.421404 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.421718 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.421792 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:03.424234 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:29:03.424235 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:29:03.424236 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:29:03.424245 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:29:03.426080 (+ 1835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:29:03.426886 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:29:03.426887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:29:03.426887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:29:03.426973 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:29:03.427358 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:29:03.427359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:29:03.427360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:29:03.427547 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:29:03.427628 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0618 23:29:03.428166 (+ 538us) server_negotiation.cc:299] Negotiation successful
0618 23:29:03.428277 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260618 23:29:03.428619 23435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:03.417376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.129:49177)
0618 23:29:03.417568 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:03.417631 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0618 23:29:03.418272 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:29:03.418551 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:29:03.418555 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:29:03.419137 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:29:03.419331 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:03.419335 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.420062 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.420063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:03.421310 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:03.421312 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.421806 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.421807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:03.421923 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:03.422476 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:29:03.422484 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:29:03.424143 (+ 1659us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:29:03.426280 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:29:03.426294 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:29:03.426297 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:29:03.426677 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:29:03.427120 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:29:03.427122 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:29:03.427122 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:29:03.427264 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:29:03.427678 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:29:03.427686 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:29:03.427873 (+ 187us) client_negotiation.cc:769] Sending connection context
0618 23:29:03.428037 (+ 164us) client_negotiation.cc:240] Negotiation successful
0618 23:29:03.428283 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":88,"threads_started":1}
I20260618 23:29:03.429584 23433 heartbeater.cc:347] Connected to a master server at 127.8.39.190:37277
WARNING: no policy specified for kudu/127.8.39.130@KRBTEST.COM; defaulting to no policy
I20260618 23:29:03.429728 23433 heartbeater.cc:464] Registering TS with master...
I20260618 23:29:03.430032 23433 heartbeater.cc:511] Master 127.8.39.190:37277 requested a full tablet report, sending...
I20260618 23:29:03.430919 23242 ts_manager.cc:194] Registered new tserver with Master: f0eff757ff2c48608da677b47879e369 (127.8.39.129:44949)
I20260618 23:29:03.431651 23242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.129@KRBTEST.COM'} at 127.8.39.129:49177
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
I20260618 23:29:03.448279 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.130:0
--local_ip_for_outbound_sockets=127.8.39.130
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:37277
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:29:03.532899 23438 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:29:03.533190 23438 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:29:03.533231 23438 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:29:03.534425 23438 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:29:03.534510 23438 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:29:03.534565 23438 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.130
I20260618 23:29:03.536055 23438 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:37277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23438
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.130
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:29:03.536335 23438 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:29:03.536624 23438 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:29:03.541898 23444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:29:03.541961 23438 server_base.cc:1034] running on GCE node
W20260618 23:29:03.541920 23443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:29:03.541920 23446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:29:03.542364 23438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:29:03.542577 23438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:29:03.543745 23438 hybrid_clock.cc:648] HybridClock initialized: now 1781825343543722 us; error 38 us; skew 500 ppm
Jun 18 23:29:03 dist-test-slave-47qt krb5kdc[23197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825343, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:29:03.546494 23438 init.cc:377] Logged in from keytab as kudu/127.8.39.130@KRBTEST.COM (short username kudu)
I20260618 23:29:03.546999 23438 webserver.cc:460] Webserver started at http://127.8.39.130:42311/ using document root <none> and password file <none>
I20260618 23:29:03.547188 23438 fs_manager.cc:359] Metadata directory not provided
I20260618 23:29:03.547228 23438 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:29:03.547315 23438 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:29:03.548300 23438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/data/instance:
uuid: "09091bdf5ff24353a21b85abb493dbb2"
format_stamp: "Formatted at 2026-06-18 23:29:03 on dist-test-slave-47qt"
server_key: "556924fd87ce3b01649349b02518e3dc"
server_key_iv: "e4ec62d0aa341e2513a59ed118b87ca6"
server_key_version: "encryptionkey@0"
I20260618 23:29:03.548611 23438 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance:
uuid: "09091bdf5ff24353a21b85abb493dbb2"
format_stamp: "Formatted at 2026-06-18 23:29:03 on dist-test-slave-47qt"
server_key: "556924fd87ce3b01649349b02518e3dc"
server_key_iv: "e4ec62d0aa341e2513a59ed118b87ca6"
server_key_version: "encryptionkey@0"
I20260618 23:29:03.549903 23438 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:29:03.550557 23453 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:29:03.550745 23438 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260618 23:29:03.550817 23438 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/wal
uuid: "09091bdf5ff24353a21b85abb493dbb2"
format_stamp: "Formatted at 2026-06-18 23:29:03 on dist-test-slave-47qt"
server_key: "556924fd87ce3b01649349b02518e3dc"
server_key_iv: "e4ec62d0aa341e2513a59ed118b87ca6"
server_key_version: "encryptionkey@0"
I20260618 23:29:03.550905 23438 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:29:03.568773 23438 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:29:03.570904 23438 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:29:03.571048 23438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:29:03.571305 23438 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:29:03.571616 23438 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:29:03.571671 23438 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:29:03.571714 23438 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:29:03.571743 23438 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:29:03.578552 23438 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.130:43995
I20260618 23:29:03.578598 23566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.130:43995 every 8 connection(s)
I20260618 23:29:03.578927 23438 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
I20260618 23:29:03.583676 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 23438
I20260618 23:29:03.583781 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance
I20260618 23:29:03.583963 8350 external_mini_cluster.cc:1428] Setting key 7f430ed7ade4112b4eb9639a0f32c9f6
Jun 18 23:29:03 dist-test-slave-47qt krb5kdc[23197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825343, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:29:03.588563 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:03.579631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:53457 (local address 127.8.39.190:37277)
0618 23:29:03.579696 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0618 23:29:03.579698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:29:03.579909 (+ 211us) server_negotiation.cc:407] Connection header received
0618 23:29:03.580382 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:29:03.580383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:29:03.580392 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:29:03.580447 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:29:03.581356 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.581859 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.582854 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.583047 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.583119 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:03.585517 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:29:03.585519 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:29:03.585519 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:29:03.585527 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:29:03.586992 (+ 1465us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:29:03.587469 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:29:03.587470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:29:03.587470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:29:03.587529 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:29:03.587805 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:29:03.587806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:29:03.587807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:29:03.587926 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:29:03.588006 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0618 23:29:03.588386 (+ 380us) server_negotiation.cc:299] Negotiation successful
0618 23:29:03.588474 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260618 23:29:03.588768 23569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:03.579611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.130:53457)
0618 23:29:03.579784 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:03.579824 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 23:29:03.580301 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:29:03.580507 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:29:03.580510 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:29:03.581074 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:29:03.581279 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:03.581283 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.581939 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.581939 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:03.582717 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:03.582719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.583131 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.583131 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:03.583224 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:03.583766 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:29:03.583774 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:29:03.585429 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:29:03.587085 (+ 1656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:29:03.587091 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:29:03.587093 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:29:03.587391 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:29:03.587611 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:29:03.587612 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:29:03.587612 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:29:03.587726 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:29:03.588056 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:29:03.588062 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:29:03.588192 (+ 130us) client_negotiation.cc:769] Sending connection context
0618 23:29:03.588348 (+ 156us) client_negotiation.cc:240] Negotiation successful
0618 23:29:03.588512 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":95,"threads_started":1}
I20260618 23:29:03.589481 23567 heartbeater.cc:347] Connected to a master server at 127.8.39.190:37277
I20260618 23:29:03.589569 23567 heartbeater.cc:464] Registering TS with master...
I20260618 23:29:03.589758 23567 heartbeater.cc:511] Master 127.8.39.190:37277 requested a full tablet report, sending...
I20260618 23:29:03.590155 23242 ts_manager.cc:194] Registered new tserver with Master: 09091bdf5ff24353a21b85abb493dbb2 (127.8.39.130:43995)
I20260618 23:29:03.590533 23242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.130@KRBTEST.COM'} at 127.8.39.130:53457
WARNING: no policy specified for kudu/127.8.39.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
I20260618 23:29:03.611347 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.131:0
--local_ip_for_outbound_sockets=127.8.39.131
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:37277
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:29:03.695025 23572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:29:03.695226 23572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:29:03.695250 23572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:29:03.696537 23572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:29:03.696586 23572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:29:03.696615 23572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.131
I20260618 23:29:03.698127 23572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:37277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.131
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:29:03.698369 23572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:29:03.698612 23572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:29:03.703933 23578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:29:03.703958 23580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:29:03.704092 23572 server_base.cc:1034] running on GCE node
W20260618 23:29:03.703929 23577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:29:03.704370 23572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:29:03.704623 23572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:29:03.705855 23572 hybrid_clock.cc:648] HybridClock initialized: now 1781825343705836 us; error 30 us; skew 500 ppm
Jun 18 23:29:03 dist-test-slave-47qt krb5kdc[23197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825343, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:29:03.711249 23572 init.cc:377] Logged in from keytab as kudu/127.8.39.131@KRBTEST.COM (short username kudu)
I20260618 23:29:03.711916 23572 webserver.cc:460] Webserver started at http://127.8.39.131:34965/ using document root <none> and password file <none>
I20260618 23:29:03.712219 23572 fs_manager.cc:359] Metadata directory not provided
I20260618 23:29:03.712301 23572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:29:03.712442 23572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:29:03.713877 23572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/data/instance:
uuid: "b2d1a5a2c767498a8b563af3d0df6d59"
format_stamp: "Formatted at 2026-06-18 23:29:03 on dist-test-slave-47qt"
server_key: "f6c7a8d5b17323f8d28798c1e395b3fa"
server_key_iv: "e70e159d50348c134592ea1a77010022"
server_key_version: "encryptionkey@0"
I20260618 23:29:03.714242 23572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance:
uuid: "b2d1a5a2c767498a8b563af3d0df6d59"
format_stamp: "Formatted at 2026-06-18 23:29:03 on dist-test-slave-47qt"
server_key: "f6c7a8d5b17323f8d28798c1e395b3fa"
server_key_iv: "e70e159d50348c134592ea1a77010022"
server_key_version: "encryptionkey@0"
I20260618 23:29:03.715708 23572 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:29:03.716638 23587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:29:03.717003 23572 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:29:03.717096 23572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/wal
uuid: "b2d1a5a2c767498a8b563af3d0df6d59"
format_stamp: "Formatted at 2026-06-18 23:29:03 on dist-test-slave-47qt"
server_key: "f6c7a8d5b17323f8d28798c1e395b3fa"
server_key_iv: "e70e159d50348c134592ea1a77010022"
server_key_version: "encryptionkey@0"
I20260618 23:29:03.717214 23572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:29:03.731031 23572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:29:03.733058 23572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:29:03.733263 23572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:29:03.733503 23572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:29:03.733829 23572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:29:03.733884 23572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:29:03.733927 23572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:29:03.733953 23572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:29:03.740546 23572 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.131:44023
I20260618 23:29:03.740585 23700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.131:44023 every 8 connection(s)
I20260618 23:29:03.740913 23572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
I20260618 23:29:03.746563 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 23572
I20260618 23:29:03.746665 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance
I20260618 23:29:03.746860 8350 external_mini_cluster.cc:1428] Setting key dced82ff9b5909d2f8adb2ebc9bf99d0
Jun 18 23:29:03 dist-test-slave-47qt krb5kdc[23197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825343, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:29:03.749975 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:03.741681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:34555 (local address 127.8.39.190:37277)
0618 23:29:03.741754 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0618 23:29:03.741756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:29:03.741933 (+ 177us) server_negotiation.cc:407] Connection header received
0618 23:29:03.742456 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:29:03.742458 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:29:03.742469 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:29:03.742533 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:29:03.743467 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.743961 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.744938 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.745177 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.745275 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:03.747509 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:29:03.747510 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:29:03.747510 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:29:03.747518 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:29:03.748746 (+ 1228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:29:03.749210 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:29:03.749211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:29:03.749211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:29:03.749267 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:29:03.749482 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:29:03.749482 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:29:03.749483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:29:03.749578 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:29:03.749627 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0618 23:29:03.749855 (+ 228us) server_negotiation.cc:299] Negotiation successful
0618 23:29:03.749913 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260618 23:29:03.750195 23703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:03.741655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.131:34555)
0618 23:29:03.741804 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:03.741852 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0618 23:29:03.742354 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:29:03.742601 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:29:03.742605 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:29:03.743185 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:29:03.743369 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:03.743374 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.744058 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.744059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:03.744839 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:03.744841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.745286 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.745287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:03.745404 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:03.745925 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:29:03.745933 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:29:03.747427 (+ 1494us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:29:03.748835 (+ 1408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:29:03.748841 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:29:03.748843 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:29:03.749060 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:29:03.749328 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:29:03.749329 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:29:03.749329 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:29:03.749425 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:29:03.749640 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:29:03.749644 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:29:03.749748 (+ 104us) client_negotiation.cc:769] Sending connection context
0618 23:29:03.749826 (+ 78us) client_negotiation.cc:240] Negotiation successful
0618 23:29:03.749939 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
I20260618 23:29:03.750671 23701 heartbeater.cc:347] Connected to a master server at 127.8.39.190:37277
I20260618 23:29:03.750771 23701 heartbeater.cc:464] Registering TS with master...
I20260618 23:29:03.750937 23701 heartbeater.cc:511] Master 127.8.39.190:37277 requested a full tablet report, sending...
I20260618 23:29:03.751259 23242 ts_manager.cc:194] Registered new tserver with Master: b2d1a5a2c767498a8b563af3d0df6d59 (127.8.39.131:44023)
I20260618 23:29:03.751596 23242 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.131@KRBTEST.COM'} at 127.8.39.131:34555
I20260618 23:29:03.758932 8350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:29:03.766513 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:03.760348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56006 (local address 127.8.39.190:37277)
0618 23:29:03.760437 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0618 23:29:03.760440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:29:03.760531 (+ 91us) server_negotiation.cc:407] Connection header received
0618 23:29:03.760628 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:29:03.760630 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:29:03.760643 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:29:03.760711 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:29:03.761338 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.761839 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.762591 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.762780 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.762855 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:03.763989 (+ 1134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:29:03.763990 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:29:03.763991 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:29:03.763998 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:29:03.765249 (+ 1251us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:29:03.765658 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:29:03.765659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:29:03.765660 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:29:03.765731 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:29:03.765950 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:29:03.765951 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:29:03.765952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:29:03.766112 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:29:03.766217 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0618 23:29:03.766330 (+ 113us) server_negotiation.cc:299] Negotiation successful
0618 23:29:03.766411 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260618 23:29:03.768639 23242 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56006:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:29:03.770450 23242 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:29:03.777433 23713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:03.772764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:44949 (local address 127.0.0.1:37958)
0618 23:29:03.772940 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:03.772991 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0618 23:29:03.773065 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:29:03.773801 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:29:03.773804 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:29:03.773807 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:29:03.773879 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:03.773881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.774928 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.774929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:03.776275 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:03.776278 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.777049 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.777050 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:03.777176 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:03.777191 (+ 15us) client_negotiation.cc:769] Sending connection context
0618 23:29:03.777287 (+ 96us) client_negotiation.cc:240] Negotiation successful
0618 23:29:03.777347 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":99,"threads_started":1}
I20260618 23:29:03.777773 23715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:03.772875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37958 (local address 127.8.39.129:44949)
0618 23:29:03.773437 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0618 23:29:03.773441 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:29:03.773451 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:29:03.773487 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:29:03.773489 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:29:03.773502 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:29:03.773958 (+ 456us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:29:03.774030 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.774809 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.776365 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.776948 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.777044 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:03.777108 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0618 23:29:03.777649 (+ 541us) server_negotiation.cc:299] Negotiation successful
0618 23:29:03.777695 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":77,"threads_started":1}
I20260618 23:29:03.778462 23714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:03.772766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:43995 (local address 127.0.0.1:54292)
0618 23:29:03.773181 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:03.773189 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0618 23:29:03.773222 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:29:03.774610 (+ 1388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:29:03.774612 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:29:03.774615 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:29:03.774664 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:03.774665 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.776148 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.776149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:03.777287 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:03.777289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.778223 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.778224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:03.778291 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:03.778296 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:29:03.778327 (+ 31us) client_negotiation.cc:240] Negotiation successful
0618 23:29:03.778376 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":398,"mutex_wait_us":4,"thread_start_us":76,"threads_started":1}
I20260618 23:29:03.778542 23716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:03.773003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60980 (local address 127.8.39.131:44023)
0618 23:29:03.773205 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0618 23:29:03.773211 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 23:29:03.773223 (+ 12us) server_negotiation.cc:407] Connection header received
0618 23:29:03.773267 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:29:03.773269 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:29:03.773286 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:29:03.773384 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:29:03.774862 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.775872 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.777287 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.778044 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.778111 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:03.778162 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0618 23:29:03.778420 (+ 258us) server_negotiation.cc:299] Negotiation successful
0618 23:29:03.778468 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":104,"threads_started":1}
I20260618 23:29:03.778455 23367 tablet_service.cc:1467] Processing CreateTablet for tablet 1cf979a835cc4db097fef520b8fabdd0 (DEFAULT_TABLE table=test-table [id=74aa77cf28f64fd3a1eda803d82d66fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:29:03.778743 23717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:03.773230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54292 (local address 127.8.39.130:43995)
0618 23:29:03.773536 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0618 23:29:03.773541 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:29:03.774342 (+ 801us) server_negotiation.cc:407] Connection header received
0618 23:29:03.774427 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:29:03.774431 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:29:03.774455 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:29:03.774587 (+ 132us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:29:03.774845 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.776025 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.777347 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.778131 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.778504 (+ 373us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:03.778575 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0618 23:29:03.778606 (+ 31us) server_negotiation.cc:299] Negotiation successful
0618 23:29:03.778659 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":84,"threads_started":1}
I20260618 23:29:03.778834 23367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf979a835cc4db097fef520b8fabdd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:29:03.778458 23712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:03.772766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:44023 (local address 127.0.0.1:60980)
0618 23:29:03.773027 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:03.773036 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:29:03.773077 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:29:03.774325 (+ 1248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:29:03.774328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:29:03.774338 (+ 10us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:29:03.774460 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:03.774462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.775989 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.775990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:03.777195 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:03.777198 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.778198 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.778199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:03.778261 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:03.778266 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:29:03.778315 (+ 49us) client_negotiation.cc:240] Negotiation successful
0618 23:29:03.778353 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":59,"threads_started":1}
I20260618 23:29:03.779527 23635 tablet_service.cc:1467] Processing CreateTablet for tablet 1cf979a835cc4db097fef520b8fabdd0 (DEFAULT_TABLE table=test-table [id=74aa77cf28f64fd3a1eda803d82d66fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:29:03.779884 23635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf979a835cc4db097fef520b8fabdd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:29:03.781540 23501 tablet_service.cc:1467] Processing CreateTablet for tablet 1cf979a835cc4db097fef520b8fabdd0 (DEFAULT_TABLE table=test-table [id=74aa77cf28f64fd3a1eda803d82d66fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:29:03.781781 23718 tablet_bootstrap.cc:492] T 1cf979a835cc4db097fef520b8fabdd0 P f0eff757ff2c48608da677b47879e369: Bootstrap starting.
I20260618 23:29:03.781872 23501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1cf979a835cc4db097fef520b8fabdd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:29:03.782353 23718 tablet_bootstrap.cc:654] T 1cf979a835cc4db097fef520b8fabdd0 P f0eff757ff2c48608da677b47879e369: Neither blocks nor log segments found. Creating new log.
I20260618 23:29:03.782774 23718 log.cc:824] T 1cf979a835cc4db097fef520b8fabdd0 P f0eff757ff2c48608da677b47879e369: Log is configured to *not* fsync() on all Append() calls
I20260618 23:29:03.783001 23719 tablet_bootstrap.cc:492] T 1cf979a835cc4db097fef520b8fabdd0 P b2d1a5a2c767498a8b563af3d0df6d59: Bootstrap starting.
I20260618 23:29:03.783612 23719 tablet_bootstrap.cc:654] T 1cf979a835cc4db097fef520b8fabdd0 P b2d1a5a2c767498a8b563af3d0df6d59: Neither blocks nor log segments found. Creating new log.
I20260618 23:29:03.783720 23718 tablet_bootstrap.cc:492] T 1cf979a835cc4db097fef520b8fabdd0 P f0eff757ff2c48608da677b47879e369: No bootstrap required, opened a new log
I20260618 23:29:03.783784 23718 ts_tablet_manager.cc:1397] T 1cf979a835cc4db097fef520b8fabdd0 P f0eff757ff2c48608da677b47879e369: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:29:03.784142 23719 log.cc:824] T 1cf979a835cc4db097fef520b8fabdd0 P b2d1a5a2c767498a8b563af3d0df6d59: Log is configured to *not* fsync() on all Append() calls
I20260618 23:29:03.784848 23722 tablet_bootstrap.cc:492] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2: Bootstrap starting.
I20260618 23:29:03.784904 23719 tablet_bootstrap.cc:492] T 1cf979a835cc4db097fef520b8fabdd0 P b2d1a5a2c767498a8b563af3d0df6d59: No bootstrap required, opened a new log
I20260618 23:29:03.784970 23719 ts_tablet_manager.cc:1397] T 1cf979a835cc4db097fef520b8fabdd0 P b2d1a5a2c767498a8b563af3d0df6d59: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:29:03.785501 23722 tablet_bootstrap.cc:654] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2: Neither blocks nor log segments found. Creating new log.
I20260618 23:29:03.786031 23722 log.cc:824] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2: Log is configured to *not* fsync() on all Append() calls
I20260618 23:29:03.786578 23718 raft_consensus.cc:348] T 1cf979a835cc4db097fef520b8fabdd0 P f0eff757ff2c48608da677b47879e369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44023 } } peers { permanent_uuid: "09091bdf5ff24353a21b85abb493dbb2" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 43995 } } peers { permanent_uuid: "f0eff757ff2c48608da677b47879e369" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 44949 } }
I20260618 23:29:03.787118 23718 raft_consensus.cc:374] T 1cf979a835cc4db097fef520b8fabdd0 P f0eff757ff2c48608da677b47879e369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:29:03.786999 23719 raft_consensus.cc:348] T 1cf979a835cc4db097fef520b8fabdd0 P b2d1a5a2c767498a8b563af3d0df6d59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44023 } } peers { permanent_uuid: "09091bdf5ff24353a21b85abb493dbb2" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 43995 } } peers { permanent_uuid: "f0eff757ff2c48608da677b47879e369" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 44949 } }
I20260618 23:29:03.787185 23718 raft_consensus.cc:729] T 1cf979a835cc4db097fef520b8fabdd0 P f0eff757ff2c48608da677b47879e369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0eff757ff2c48608da677b47879e369, State: Initialized, Role: FOLLOWER
I20260618 23:29:03.787196 23719 raft_consensus.cc:374] T 1cf979a835cc4db097fef520b8fabdd0 P b2d1a5a2c767498a8b563af3d0df6d59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:29:03.787235 23719 raft_consensus.cc:729] T 1cf979a835cc4db097fef520b8fabdd0 P b2d1a5a2c767498a8b563af3d0df6d59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d1a5a2c767498a8b563af3d0df6d59, State: Initialized, Role: FOLLOWER
I20260618 23:29:03.787289 23722 tablet_bootstrap.cc:492] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2: No bootstrap required, opened a new log
I20260618 23:29:03.787365 23719 consensus_queue.cc:260] T 1cf979a835cc4db097fef520b8fabdd0 P b2d1a5a2c767498a8b563af3d0df6d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44023 } } peers { permanent_uuid: "09091bdf5ff24353a21b85abb493dbb2" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 43995 } } peers { permanent_uuid: "f0eff757ff2c48608da677b47879e369" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 44949 } }
I20260618 23:29:03.787446 23722 ts_tablet_manager.cc:1397] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 23:29:03.787417 23718 consensus_queue.cc:260] T 1cf979a835cc4db097fef520b8fabdd0 P f0eff757ff2c48608da677b47879e369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44023 } } peers { permanent_uuid: "09091bdf5ff24353a21b85abb493dbb2" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 43995 } } peers { permanent_uuid: "f0eff757ff2c48608da677b47879e369" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 44949 } }
I20260618 23:29:03.787670 23719 ts_tablet_manager.cc:1428] T 1cf979a835cc4db097fef520b8fabdd0 P b2d1a5a2c767498a8b563af3d0df6d59: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:29:03.787823 23701 heartbeater.cc:503] Master 127.8.39.190:37277 was elected leader, sending a full tablet report...
I20260618 23:29:03.787832 23718 ts_tablet_manager.cc:1428] T 1cf979a835cc4db097fef520b8fabdd0 P f0eff757ff2c48608da677b47879e369: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260618 23:29:03.788017 23433 heartbeater.cc:503] Master 127.8.39.190:37277 was elected leader, sending a full tablet report...
I20260618 23:29:03.790212 23722 raft_consensus.cc:348] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44023 } } peers { permanent_uuid: "09091bdf5ff24353a21b85abb493dbb2" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 43995 } } peers { permanent_uuid: "f0eff757ff2c48608da677b47879e369" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 44949 } }
I20260618 23:29:03.790362 23722 raft_consensus.cc:374] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:29:03.790385 23722 raft_consensus.cc:729] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09091bdf5ff24353a21b85abb493dbb2, State: Initialized, Role: FOLLOWER
I20260618 23:29:03.790524 23722 consensus_queue.cc:260] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44023 } } peers { permanent_uuid: "09091bdf5ff24353a21b85abb493dbb2" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 43995 } } peers { permanent_uuid: "f0eff757ff2c48608da677b47879e369" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 44949 } }
I20260618 23:29:03.790815 23722 ts_tablet_manager.cc:1428] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:29:03.790865 23567 heartbeater.cc:503] Master 127.8.39.190:37277 was elected leader, sending a full tablet report...
I20260618 23:29:03.801301 23726 raft_consensus.cc:482] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:29:03.801450 23726 raft_consensus.cc:504] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44023 } } peers { permanent_uuid: "09091bdf5ff24353a21b85abb493dbb2" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 43995 } } peers { permanent_uuid: "f0eff757ff2c48608da677b47879e369" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 44949 } }
I20260618 23:29:03.801852 23726 leader_election.cc:288] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2d1a5a2c767498a8b563af3d0df6d59 (127.8.39.131:44023), f0eff757ff2c48608da677b47879e369 (127.8.39.129:44949)
I20260618 23:29:03.805003 23569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:03.802033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:44949 (local address 127.8.39.130:47407)
0618 23:29:03.802140 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:03.802153 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:29:03.802207 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:29:03.802404 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:29:03.802405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:29:03.802407 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:29:03.802491 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:03.802493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.803195 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.803196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:03.804105 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:03.804107 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.804757 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.804758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:03.804839 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:03.804844 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:29:03.804900 (+ 56us) client_negotiation.cc:240] Negotiation successful
0618 23:29:03.804948 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89}
I20260618 23:29:03.805068 23715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:03.802091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:47407 (local address 127.8.39.129:44949)
0618 23:29:03.802143 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0618 23:29:03.802146 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:29:03.802192 (+ 46us) server_negotiation.cc:407] Connection header received
0618 23:29:03.802268 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:29:03.802269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:29:03.802281 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:29:03.802350 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:29:03.802568 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.803115 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.804183 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.804679 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.804737 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:03.804745 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0618 23:29:03.804937 (+ 192us) server_negotiation.cc:299] Negotiation successful
0618 23:29:03.804984 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260618 23:29:03.805269 23716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:03.802182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:38395 (local address 127.8.39.131:44023)
0618 23:29:03.802237 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0618 23:29:03.802239 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:29:03.802399 (+ 160us) server_negotiation.cc:407] Connection header received
0618 23:29:03.802428 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:29:03.802429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:29:03.802438 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:29:03.802508 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:29:03.802690 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.803168 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.804282 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.804803 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.804882 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:03.804891 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0618 23:29:03.805125 (+ 234us) server_negotiation.cc:299] Negotiation successful
0618 23:29:03.805190 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260618 23:29:03.805519 23727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:03.802045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:44023 (local address 127.8.39.130:38395)
0618 23:29:03.802311 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:03.802321 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:29:03.802375 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:29:03.802515 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:29:03.802516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:29:03.802517 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:29:03.802625 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:03.802626 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.803298 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.803299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:03.804196 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:03.804198 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.804940 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.804942 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:03.805015 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:03.805020 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:29:03.805380 (+ 360us) client_negotiation.cc:240] Negotiation successful
0618 23:29:03.805435 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":43,"thread_start_us":108,"threads_started":1}
I20260618 23:29:03.805629 23387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf979a835cc4db097fef520b8fabdd0" candidate_uuid: "09091bdf5ff24353a21b85abb493dbb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0eff757ff2c48608da677b47879e369" is_pre_election: true
I20260618 23:29:03.805922 23387 raft_consensus.cc:2454] T 1cf979a835cc4db097fef520b8fabdd0 P f0eff757ff2c48608da677b47879e369 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09091bdf5ff24353a21b85abb493dbb2 in term 0.
I20260618 23:29:03.806237 23456 leader_election.cc:302] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09091bdf5ff24353a21b85abb493dbb2, f0eff757ff2c48608da677b47879e369; no voters:
I20260618 23:29:03.806370 23726 raft_consensus.cc:2781] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:29:03.806452 23726 raft_consensus.cc:482] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:29:03.806481 23726 raft_consensus.cc:3037] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:29:03.806509 23655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf979a835cc4db097fef520b8fabdd0" candidate_uuid: "09091bdf5ff24353a21b85abb493dbb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" is_pre_election: true
I20260618 23:29:03.806694 23655 raft_consensus.cc:2454] T 1cf979a835cc4db097fef520b8fabdd0 P b2d1a5a2c767498a8b563af3d0df6d59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09091bdf5ff24353a21b85abb493dbb2 in term 0.
I20260618 23:29:03.807425 23726 raft_consensus.cc:504] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44023 } } peers { permanent_uuid: "09091bdf5ff24353a21b85abb493dbb2" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 43995 } } peers { permanent_uuid: "f0eff757ff2c48608da677b47879e369" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 44949 } }
I20260618 23:29:03.807602 23726 leader_election.cc:288] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2 [CANDIDATE]: Term 1 election: Requested vote from peers b2d1a5a2c767498a8b563af3d0df6d59 (127.8.39.131:44023), f0eff757ff2c48608da677b47879e369 (127.8.39.129:44949)
I20260618 23:29:03.807755 23387 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf979a835cc4db097fef520b8fabdd0" candidate_uuid: "09091bdf5ff24353a21b85abb493dbb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0eff757ff2c48608da677b47879e369"
I20260618 23:29:03.807879 23387 raft_consensus.cc:3037] T 1cf979a835cc4db097fef520b8fabdd0 P f0eff757ff2c48608da677b47879e369 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:29:03.808207 23655 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1cf979a835cc4db097fef520b8fabdd0" candidate_uuid: "09091bdf5ff24353a21b85abb493dbb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2d1a5a2c767498a8b563af3d0df6d59"
I20260618 23:29:03.808313 23655 raft_consensus.cc:3037] T 1cf979a835cc4db097fef520b8fabdd0 P b2d1a5a2c767498a8b563af3d0df6d59 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:29:03.808661 23387 raft_consensus.cc:2454] T 1cf979a835cc4db097fef520b8fabdd0 P f0eff757ff2c48608da677b47879e369 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09091bdf5ff24353a21b85abb493dbb2 in term 1.
I20260618 23:29:03.808825 23456 leader_election.cc:302] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09091bdf5ff24353a21b85abb493dbb2, f0eff757ff2c48608da677b47879e369; no voters:
I20260618 23:29:03.808907 23726 raft_consensus.cc:2781] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:29:03.809134 23726 raft_consensus.cc:686] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2 [term 1 LEADER]: Becoming Leader. State: Replica: 09091bdf5ff24353a21b85abb493dbb2, State: Running, Role: LEADER
I20260618 23:29:03.809234 23726 consensus_queue.cc:237] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44023 } } peers { permanent_uuid: "09091bdf5ff24353a21b85abb493dbb2" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 43995 } } peers { permanent_uuid: "f0eff757ff2c48608da677b47879e369" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 44949 } }
I20260618 23:29:03.809284 23655 raft_consensus.cc:2454] T 1cf979a835cc4db097fef520b8fabdd0 P b2d1a5a2c767498a8b563af3d0df6d59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09091bdf5ff24353a21b85abb493dbb2 in term 1.
I20260618 23:29:03.810127 23242 catalog_manager.cc:5495] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09091bdf5ff24353a21b85abb493dbb2 (127.8.39.130). New cstate: current_term: 1 leader_uuid: "09091bdf5ff24353a21b85abb493dbb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09091bdf5ff24353a21b85abb493dbb2" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 43995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f0eff757ff2c48608da677b47879e369" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 44949 } health_report { overall_health: UNKNOWN } } }
I20260618 23:29:03.824923 23717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:03.821702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54294 (local address 127.8.39.130:43995)
0618 23:29:03.821893 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0618 23:29:03.821896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:29:03.821910 (+ 14us) server_negotiation.cc:407] Connection header received
0618 23:29:03.821970 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:29:03.821971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:29:03.822003 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:29:03.822177 (+ 174us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:29:03.822417 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.822938 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.823994 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:03.824236 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:03.824327 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:03.824528 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:29:03.824641 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:29:03.824743 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0618 23:29:03.824786 (+ 43us) server_negotiation.cc:299] Negotiation successful
0618 23:29:03.824811 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149}
I20260618 23:29:03.826781 23655 raft_consensus.cc:1261] T 1cf979a835cc4db097fef520b8fabdd0 P b2d1a5a2c767498a8b563af3d0df6d59 [term 1 FOLLOWER]: Refusing update from remote peer 09091bdf5ff24353a21b85abb493dbb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:29:03.826805 23387 raft_consensus.cc:1261] T 1cf979a835cc4db097fef520b8fabdd0 P f0eff757ff2c48608da677b47879e369 [term 1 FOLLOWER]: Refusing update from remote peer 09091bdf5ff24353a21b85abb493dbb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:29:03.827116 23728 consensus_queue.cc:1035] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:29:03.827250 23726 consensus_queue.cc:1035] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0eff757ff2c48608da677b47879e369" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 44949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260618 23:29:03.829391 23568 tablet.cc:2369] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:29:03.830214 23731 mvcc.cc:205] Tried to move back new op lower bound from 7298356608311668736 to 7298356608243150848. Current Snapshot: MvccSnapshot[applied={T|T < 7298356608311668736}]
I20260618 23:29:03.833226 23241 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:29:03.833326 23241 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56006:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:29:03.834342 23241 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f: Sending DeleteTablet for 3 replicas of tablet 1cf979a835cc4db097fef520b8fabdd0
I20260618 23:29:03.834702 23501 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cf979a835cc4db097fef520b8fabdd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:29:03 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:54292
I20260618 23:29:03.834715 23635 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cf979a835cc4db097fef520b8fabdd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:29:03 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:60980
I20260618 23:29:03.834944 23367 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cf979a835cc4db097fef520b8fabdd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:29:03 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:37958
I20260618 23:29:03.834981 23741 tablet_replica.cc:330] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2: stopping tablet replica
I20260618 23:29:03.834990 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 23304
I20260618 23:29:03.835628 23741 raft_consensus.cc:2229] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2 [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:29:03.835757 23741 raft_consensus.cc:2258] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:29:03.835850 23742 tablet_replica.cc:330] T 1cf979a835cc4db097fef520b8fabdd0 P b2d1a5a2c767498a8b563af3d0df6d59: stopping tablet replica
I20260618 23:29:03.835938 23742 raft_consensus.cc:2229] T 1cf979a835cc4db097fef520b8fabdd0 P b2d1a5a2c767498a8b563af3d0df6d59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:29:03.836030 23742 raft_consensus.cc:2258] T 1cf979a835cc4db097fef520b8fabdd0 P b2d1a5a2c767498a8b563af3d0df6d59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:29:03.836359 23741 ts_tablet_manager.cc:1905] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:29:03.836418 23742 ts_tablet_manager.cc:1905] T 1cf979a835cc4db097fef520b8fabdd0 P b2d1a5a2c767498a8b563af3d0df6d59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:29:03.838099 23742 ts_tablet_manager.cc:1918] T 1cf979a835cc4db097fef520b8fabdd0 P b2d1a5a2c767498a8b563af3d0df6d59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:29:03.838121 23741 ts_tablet_manager.cc:1918] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:29:03.838165 23742 log.cc:1196] T 1cf979a835cc4db097fef520b8fabdd0 P b2d1a5a2c767498a8b563af3d0df6d59: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/wal/wals/1cf979a835cc4db097fef520b8fabdd0
I20260618 23:29:03.838173 23741 log.cc:1196] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/wal/wals/1cf979a835cc4db097fef520b8fabdd0
I20260618 23:29:03.838446 23741 ts_tablet_manager.cc:1939] T 1cf979a835cc4db097fef520b8fabdd0 P 09091bdf5ff24353a21b85abb493dbb2: Deleting consensus metadata
I20260618 23:29:03.838446 23742 ts_tablet_manager.cc:1939] T 1cf979a835cc4db097fef520b8fabdd0 P b2d1a5a2c767498a8b563af3d0df6d59: Deleting consensus metadata
I20260618 23:29:03.838820 23227 catalog_manager.cc:4841] TS 09091bdf5ff24353a21b85abb493dbb2 (127.8.39.130:43995): tablet 1cf979a835cc4db097fef520b8fabdd0 (table test-table [id=74aa77cf28f64fd3a1eda803d82d66fa]) successfully deleted
I20260618 23:29:03.838833 23229 catalog_manager.cc:4841] TS b2d1a5a2c767498a8b563af3d0df6d59 (127.8.39.131:44023): tablet 1cf979a835cc4db097fef520b8fabdd0 (table test-table [id=74aa77cf28f64fd3a1eda803d82d66fa]) successfully deleted
W20260618 23:29:03.842898 23228 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.39.129:44949 (error 108)
I20260618 23:29:03.843176 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 23438
I20260618 23:29:03.843856 23712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:03.843295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:44949 (local address 127.0.0.1:37966)
0618 23:29:03.843780 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:03.843817 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.129:44949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464}
W20260618 23:29:03.845337 23228 catalog_manager.cc:4584] TS f0eff757ff2c48608da677b47879e369 (127.8.39.129:44949): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1cf979a835cc4db097fef520b8fabdd0: Network error: Client connection negotiation failed: client connection to 127.8.39.129:44949: connect: Connection refused (error 111)
I20260618 23:29:03.848194 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 23572
I20260618 23:29:03.853287 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 23210
I20260618 23:29:23.859115 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:37277
--webserver_interface=127.8.39.190
--webserver_port=40913
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:37277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:29:23.941823 23745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:29:23.942040 23745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:29:23.942065 23745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:29:23.943253 23745 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260618 23:29:23.943295 23745 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260618 23:29:23.943308 23745 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260618 23:29:23.943320 23745 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:29:23.943333 23745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:29:23.943347 23745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:29:23.943401 23745 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:29:23.943418 23745 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:29:23.944892 23745 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:37277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:37277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=40913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:29:23.945156 23745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:29:23.945407 23745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:29:23.955046 23754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:29:23.955129 23745 server_base.cc:1034] running on GCE node
W20260618 23:29:23.955168 23751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:29:23.955046 23752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:29:23.955633 23745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:29:23.955958 23745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:29:23.957175 23745 hybrid_clock.cc:648] HybridClock initialized: now 1781825363957155 us; error 38 us; skew 500 ppm
Jun 18 23:29:23 dist-test-slave-47qt krb5kdc[23197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825363, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:29:23.964743 23745 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:29:23.965312 23745 webserver.cc:460] Webserver started at http://127.8.39.190:40913/ using document root <none> and password file <none>
I20260618 23:29:23.965577 23745 fs_manager.cc:359] Metadata directory not provided
I20260618 23:29:23.965660 23745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:29:23.967113 23745 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:29:23.967937 23761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:29:23.968351 23745 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:29:23.968483 23745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "28d09a3f812648c487d8fb8164065f7f"
format_stamp: "Formatted at 2026-06-18 23:29:03 on dist-test-slave-47qt"
server_key: "06db7f3f59455f9cbcd91986c295560a"
server_key_iv: "39715f061ebe9dc819ce3b157cfd1ba9"
server_key_version: "encryptionkey@0"
I20260618 23:29:23.968947 23745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:29:23.977051 23745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:29:23.979528 23745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:29:23.979699 23745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:29:23.984375 23745 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:37277
I20260618 23:29:23.984403 23813 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:37277 every 8 connection(s)
I20260618 23:29:23.984808 23745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:29:23.986359 23814 sys_catalog.cc:263] Verifying existing consensus state
I20260618 23:29:23.987035 23814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f: Bootstrap starting.
I20260618 23:29:23.994926 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 23745
I20260618 23:29:23.995251 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.129:44949
--local_ip_for_outbound_sockets=127.8.39.129
--tserver_master_addrs=127.8.39.190:37277
--webserver_port=44159
--webserver_interface=127.8.39.129
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:29:23.997712 23814 log.cc:824] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f: Log is configured to *not* fsync() on all Append() calls
I20260618 23:29:23.999825 23814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 23:29:24.000234 23814 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f: Bootstrap complete.
I20260618 23:29:24.002635 23814 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d09a3f812648c487d8fb8164065f7f" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 37277 } }
I20260618 23:29:24.002874 23814 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28d09a3f812648c487d8fb8164065f7f, State: Initialized, Role: FOLLOWER
I20260618 23:29:24.003049 23814 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d09a3f812648c487d8fb8164065f7f" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 37277 } }
I20260618 23:29:24.003125 23814 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:29:24.003182 23814 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:29:24.003229 23814 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [term 1 FOLLOWER]: Advancing to term 2
I20260618 23:29:24.004192 23814 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d09a3f812648c487d8fb8164065f7f" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 37277 } }
I20260618 23:29:24.004309 23814 leader_election.cc:302] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28d09a3f812648c487d8fb8164065f7f; no voters:
I20260618 23:29:24.004495 23814 leader_election.cc:288] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260618 23:29:24.004527 23818 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [term 2 FOLLOWER]: Leader election won for term 2
I20260618 23:29:24.004637 23818 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [term 2 LEADER]: Becoming Leader. State: Replica: 28d09a3f812648c487d8fb8164065f7f, State: Running, Role: LEADER
I20260618 23:29:24.004719 23818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d09a3f812648c487d8fb8164065f7f" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 37277 } }
I20260618 23:29:24.004755 23814 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:29:24.004983 23818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28d09a3f812648c487d8fb8164065f7f. Latest consensus state: current_term: 2 leader_uuid: "28d09a3f812648c487d8fb8164065f7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d09a3f812648c487d8fb8164065f7f" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 37277 } } }
I20260618 23:29:24.005052 23818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [sys.catalog]: This master's current role is: LEADER
I20260618 23:29:24.005200 23819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "28d09a3f812648c487d8fb8164065f7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d09a3f812648c487d8fb8164065f7f" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 37277 } } }
I20260618 23:29:24.005275 23819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [sys.catalog]: This master's current role is: LEADER
I20260618 23:29:24.005582 23822 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:29:24.006336 23822 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:29:24.006438 23822 catalog_manager.cc:1259] Loaded cluster ID: 9cc7d5021b2f4aed9925d525df5fda1c
I20260618 23:29:24.006474 23822 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:29:24.007898 23822 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:29:24.008080 23822 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f: Loaded TSK: 0
I20260618 23:29:24.017172 23822 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f: Generated new TSK 1
I20260618 23:29:24.017268 23822 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260618 23:29:24.083168 23816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:29:24.083384 23816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:29:24.083415 23816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:29:24.084702 23816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:29:24.084748 23816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:29:24.084775 23816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.129
I20260618 23:29:24.086328 23816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.129:44949
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.39.129
--webserver_port=44159
--tserver_master_addrs=127.8.39.190:37277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.129
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:29:24.086580 23816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:29:24.086831 23816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:29:24.092653 23844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:29:24.092654 23842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:29:24.092643 23841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:29:24.092960 23816 server_base.cc:1034] running on GCE node
I20260618 23:29:24.093184 23816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:29:24.093441 23816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:29:24.094605 23816 hybrid_clock.cc:648] HybridClock initialized: now 1781825364094584 us; error 40 us; skew 500 ppm
Jun 18 23:29:24 dist-test-slave-47qt krb5kdc[23197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825364, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:29:24.097153 23816 init.cc:377] Logged in from keytab as kudu/127.8.39.129@KRBTEST.COM (short username kudu)
I20260618 23:29:24.097683 23816 webserver.cc:460] Webserver started at http://127.8.39.129:44159/ using document root <none> and password file <none>
I20260618 23:29:24.097935 23816 fs_manager.cc:359] Metadata directory not provided
I20260618 23:29:24.097991 23816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:29:24.099366 23816 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:29:24.100071 23851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:29:24.100288 23816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:29:24.100389 23816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/wal
uuid: "f0eff757ff2c48608da677b47879e369"
format_stamp: "Formatted at 2026-06-18 23:29:03 on dist-test-slave-47qt"
server_key: "1acb970c37529997eee9ea3ba9447865"
server_key_iv: "aaf8994287dcaa88f275596a9d1ac3e0"
server_key_version: "encryptionkey@0"
I20260618 23:29:24.100801 23816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:29:24.156801 23816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:29:24.159312 23816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:29:24.159478 23816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:29:24.159721 23816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:29:24.160136 23858 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260618 23:29:24.161813 23816 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260618 23:29:24.161971 23816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260618 23:29:24.162024 23816 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260618 23:29:24.162810 23816 ts_tablet_manager.cc:610] Registered 1 tablets
I20260618 23:29:24.162875 23816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260618 23:29:24.163000 23858 tablet_bootstrap.cc:492] T 1cf979a835cc4db097fef520b8fabdd0 P f0eff757ff2c48608da677b47879e369: Bootstrap starting.
I20260618 23:29:24.169567 23816 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.129:44949
I20260618 23:29:24.169670 23965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.129:44949 every 8 connection(s)
I20260618 23:29:24.170039 23816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
I20260618 23:29:24.171262 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 23816
I20260618 23:29:24.171566 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.130:43995
--local_ip_for_outbound_sockets=127.8.39.130
--tserver_master_addrs=127.8.39.190:37277
--webserver_port=42311
--webserver_interface=127.8.39.130
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:29:24.174161 23858 log.cc:824] T 1cf979a835cc4db097fef520b8fabdd0 P f0eff757ff2c48608da677b47879e369: Log is configured to *not* fsync() on all Append() calls
I20260618 23:29:24.177484 23858 tablet_bootstrap.cc:492] T 1cf979a835cc4db097fef520b8fabdd0 P f0eff757ff2c48608da677b47879e369: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 23:29:24.177822 23858 tablet_bootstrap.cc:492] T 1cf979a835cc4db097fef520b8fabdd0 P f0eff757ff2c48608da677b47879e369: Bootstrap complete.
I20260618 23:29:24.178925 23858 ts_tablet_manager.cc:1397] T 1cf979a835cc4db097fef520b8fabdd0 P f0eff757ff2c48608da677b47879e369: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.005s
I20260618 23:29:24.180377 23858 raft_consensus.cc:348] T 1cf979a835cc4db097fef520b8fabdd0 P f0eff757ff2c48608da677b47879e369 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44023 } } peers { permanent_uuid: "09091bdf5ff24353a21b85abb493dbb2" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 43995 } } peers { permanent_uuid: "f0eff757ff2c48608da677b47879e369" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 44949 } }
I20260618 23:29:24.180632 23858 raft_consensus.cc:729] T 1cf979a835cc4db097fef520b8fabdd0 P f0eff757ff2c48608da677b47879e369 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0eff757ff2c48608da677b47879e369, State: Initialized, Role: FOLLOWER
I20260618 23:29:24.180765 23858 consensus_queue.cc:260] T 1cf979a835cc4db097fef520b8fabdd0 P f0eff757ff2c48608da677b47879e369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44023 } } peers { permanent_uuid: "09091bdf5ff24353a21b85abb493dbb2" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 43995 } } peers { permanent_uuid: "f0eff757ff2c48608da677b47879e369" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 44949 } }
Jun 18 23:29:24 dist-test-slave-47qt krb5kdc[23197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825364, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:29:24.181069 23858 ts_tablet_manager.cc:1428] T 1cf979a835cc4db097fef520b8fabdd0 P f0eff757ff2c48608da677b47879e369: Time spent starting tablet: real 0.002s user 0.003s sys 0.001s
I20260618 23:29:24.185058 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:24.170921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:51747 (local address 127.8.39.190:37277)
0618 23:29:24.173220 (+ 2299us) server_negotiation.cc:206] Beginning negotiation
0618 23:29:24.173231 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0618 23:29:24.173281 (+ 50us) server_negotiation.cc:407] Connection header received
0618 23:29:24.173434 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:29:24.173442 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:29:24.173520 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:29:24.173686 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:29:24.174468 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.175898 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.177777 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.178466 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.178611 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:24.181542 (+ 2931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:29:24.181546 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:29:24.181546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:29:24.181556 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:29:24.183202 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:29:24.183696 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:29:24.183698 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:29:24.183699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:29:24.183783 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:29:24.184139 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:29:24.184140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:29:24.184140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:29:24.184367 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:29:24.184441 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 23:29:24.184788 (+ 347us) server_negotiation.cc:299] Negotiation successful
0618 23:29:24.184925 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2244,"thread_start_us":115,"threads_started":1}
I20260618 23:29:24.185071 23968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:24.170814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.129:51747)
0618 23:29:24.171051 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:24.171098 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0618 23:29:24.171679 (+ 581us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:29:24.173729 (+ 2050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:29:24.173733 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:29:24.174114 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:29:24.174341 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:24.174347 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.176029 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.176030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:24.177452 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:24.177455 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.178597 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.178599 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:24.178749 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:24.179446 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:29:24.179456 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:29:24.181430 (+ 1974us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:29:24.183291 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:29:24.183299 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:29:24.183301 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:29:24.183610 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:29:24.183851 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:29:24.183852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:29:24.183852 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:29:24.183982 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:29:24.184443 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:29:24.184448 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:29:24.184595 (+ 147us) client_negotiation.cc:769] Sending connection context
0618 23:29:24.184712 (+ 117us) client_negotiation.cc:240] Negotiation successful
0618 23:29:24.184878 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":124,"threads_started":1}
I20260618 23:29:24.185827 23966 heartbeater.cc:347] Connected to a master server at 127.8.39.190:37277
I20260618 23:29:24.185953 23966 heartbeater.cc:464] Registering TS with master...
I20260618 23:29:24.186142 23966 heartbeater.cc:511] Master 127.8.39.190:37277 requested a full tablet report, sending...
I20260618 23:29:24.186820 23778 ts_manager.cc:194] Registered new tserver with Master: f0eff757ff2c48608da677b47879e369 (127.8.39.129:44949)
I20260618 23:29:24.188189 23778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.129@KRBTEST.COM'} at 127.8.39.129:51747
I20260618 23:29:24.189181 23966 heartbeater.cc:503] Master 127.8.39.190:37277 was elected leader, sending a full tablet report...
Jun 18 23:29:24 dist-test-slave-47qt krb5kdc[23197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825363, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for kudu/127.8.39.129@KRBTEST.COM
I20260618 23:29:24.197521 23973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:24.187608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:44949 (local address 127.0.0.1:48904)
0618 23:29:24.188486 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:24.188518 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0618 23:29:24.188582 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:29:24.188852 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:29:24.188859 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:29:24.189211 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:29:24.189271 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:24.189274 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.190677 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.190678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:24.191289 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:24.191291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.191906 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.191907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:24.191970 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:24.192478 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:29:24.192482 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:29:24.193984 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:29:24.195971 (+ 1987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:29:24.195978 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:29:24.195980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:29:24.196184 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:29:24.196402 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:29:24.196403 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:29:24.196403 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:29:24.196459 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:29:24.196759 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:29:24.196765 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:29:24.196959 (+ 194us) client_negotiation.cc:769] Sending connection context
0618 23:29:24.197308 (+ 349us) client_negotiation.cc:240] Negotiation successful
0618 23:29:24.197453 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"thread_start_us":100,"threads_started":1}
I20260618 23:29:24.197532 23974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:24.187658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48904 (local address 127.8.39.129:44949)
0618 23:29:24.188703 (+ 1045us) server_negotiation.cc:206] Beginning negotiation
0618 23:29:24.188707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:29:24.188716 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:29:24.188750 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:29:24.188752 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:29:24.188766 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:29:24.188829 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:29:24.189370 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.190585 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.191371 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.191816 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.191898 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:24.194073 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:29:24.194080 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:29:24.194081 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:29:24.194088 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:29:24.195862 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:29:24.196267 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:29:24.196270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:29:24.196270 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:29:24.196332 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:29:24.196518 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:29:24.196519 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:29:24.196519 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:29:24.196668 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:29:24.196751 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0618 23:29:24.197358 (+ 607us) server_negotiation.cc:299] Negotiation successful
0618 23:29:24.197443 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1014,"thread_start_us":60,"threads_started":1}
I20260618 23:29:24.197968 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet 1cf979a835cc4db097fef520b8fabdd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:29:03 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:48904
I20260618 23:29:24.198509 23975 tablet_replica.cc:330] stopping tablet replica
I20260618 23:29:24.198621 23975 raft_consensus.cc:2229] T 1cf979a835cc4db097fef520b8fabdd0 P f0eff757ff2c48608da677b47879e369 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:29:24.198709 23975 raft_consensus.cc:2258] T 1cf979a835cc4db097fef520b8fabdd0 P f0eff757ff2c48608da677b47879e369 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:29:24.199002 23975 ts_tablet_manager.cc:1905] T 1cf979a835cc4db097fef520b8fabdd0 P f0eff757ff2c48608da677b47879e369: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:29:24.200996 23975 ts_tablet_manager.cc:1918] T 1cf979a835cc4db097fef520b8fabdd0 P f0eff757ff2c48608da677b47879e369: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:29:24.201084 23975 log.cc:1196] T 1cf979a835cc4db097fef520b8fabdd0 P f0eff757ff2c48608da677b47879e369: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/wal/wals/1cf979a835cc4db097fef520b8fabdd0
I20260618 23:29:24.201372 23975 ts_tablet_manager.cc:1939] T 1cf979a835cc4db097fef520b8fabdd0 P f0eff757ff2c48608da677b47879e369: Deleting consensus metadata
I20260618 23:29:24.201915 23764 catalog_manager.cc:4841] TS f0eff757ff2c48608da677b47879e369 (127.8.39.129:44949): tablet 1cf979a835cc4db097fef520b8fabdd0 (table test-table [id=74aa77cf28f64fd3a1eda803d82d66fa]) successfully deleted
W20260618 23:29:24.265004 23970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:29:24.265297 23970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:29:24.265340 23970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:29:24.266525 23970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:29:24.266594 23970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:29:24.266639 23970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.130
I20260618 23:29:24.268270 23970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.130:43995
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.39.130
--webserver_port=42311
--tserver_master_addrs=127.8.39.190:37277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.130
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:29:24.268537 23970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:29:24.268785 23970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:29:24.274202 23980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:29:24.274190 23983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:29:24.274528 23970 server_base.cc:1034] running on GCE node
W20260618 23:29:24.274245 23981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:29:24.274909 23970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:29:24.275240 23970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:29:24.276453 23970 hybrid_clock.cc:648] HybridClock initialized: now 1781825364276453 us; error 46 us; skew 500 ppm
Jun 18 23:29:24 dist-test-slave-47qt krb5kdc[23197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825364, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:29:24.279476 23970 init.cc:377] Logged in from keytab as kudu/127.8.39.130@KRBTEST.COM (short username kudu)
I20260618 23:29:24.279947 23970 webserver.cc:460] Webserver started at http://127.8.39.130:42311/ using document root <none> and password file <none>
I20260618 23:29:24.280130 23970 fs_manager.cc:359] Metadata directory not provided
I20260618 23:29:24.280180 23970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:29:24.281605 23970 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260618 23:29:24.282204 23990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:29:24.282390 23970 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:29:24.282473 23970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/wal
uuid: "09091bdf5ff24353a21b85abb493dbb2"
format_stamp: "Formatted at 2026-06-18 23:29:03 on dist-test-slave-47qt"
server_key: "556924fd87ce3b01649349b02518e3dc"
server_key_iv: "e4ec62d0aa341e2513a59ed118b87ca6"
server_key_version: "encryptionkey@0"
I20260618 23:29:24.282801 23970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:29:24.296670 23970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:29:24.298939 23970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:29:24.299058 23970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:29:24.299243 23970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:29:24.299499 23970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:29:24.299531 23970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:29:24.299554 23970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:29:24.299566 23970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:29:24.305531 23970 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.130:43995
I20260618 23:29:24.305598 24103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.130:43995 every 8 connection(s)
I20260618 23:29:24.305966 23970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
I20260618 23:29:24.308108 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 23970
I20260618 23:29:24.308372 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.131:44023
--local_ip_for_outbound_sockets=127.8.39.131
--tserver_master_addrs=127.8.39.190:37277
--webserver_port=34965
--webserver_interface=127.8.39.131
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 18 23:29:24 dist-test-slave-47qt krb5kdc[23197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825364, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:29:24.317664 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:24.306725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:43435 (local address 127.8.39.190:37277)
0618 23:29:24.306776 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0618 23:29:24.306778 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:29:24.307055 (+ 277us) server_negotiation.cc:407] Connection header received
0618 23:29:24.307578 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:29:24.307579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:29:24.307589 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:29:24.307647 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:29:24.308673 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.309501 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.310758 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.311041 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.311138 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:24.314185 (+ 3047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:29:24.314187 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:29:24.314187 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:29:24.314197 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:29:24.315762 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:29:24.316257 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:29:24.316258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:29:24.316258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:29:24.316316 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:29:24.316869 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:29:24.316871 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:29:24.316871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:29:24.317005 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:29:24.317093 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0618 23:29:24.317463 (+ 370us) server_negotiation.cc:299] Negotiation successful
0618 23:29:24.317567 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260618 23:29:24.318014 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:24.306685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.130:43435)
0618 23:29:24.306889 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:24.306954 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0618 23:29:24.307490 (+ 536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:29:24.307693 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:29:24.307696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:29:24.308311 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:29:24.308581 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:24.308587 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.309615 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.309616 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:24.310470 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:24.310473 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.311168 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.311170 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:24.311364 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:24.312065 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:29:24.312076 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:29:24.314056 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:29:24.315895 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:29:24.315903 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:29:24.315905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:29:24.316162 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:29:24.316596 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:29:24.316597 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:29:24.316597 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:29:24.316739 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:29:24.317085 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:29:24.317092 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:29:24.317292 (+ 200us) client_negotiation.cc:769] Sending connection context
0618 23:29:24.317443 (+ 151us) client_negotiation.cc:240] Negotiation successful
0618 23:29:24.317624 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":102,"threads_started":1}
I20260618 23:29:24.318864 24104 heartbeater.cc:347] Connected to a master server at 127.8.39.190:37277
I20260618 23:29:24.319082 24104 heartbeater.cc:464] Registering TS with master...
I20260618 23:29:24.319334 24104 heartbeater.cc:511] Master 127.8.39.190:37277 requested a full tablet report, sending...
I20260618 23:29:24.319916 23778 ts_manager.cc:194] Registered new tserver with Master: 09091bdf5ff24353a21b85abb493dbb2 (127.8.39.130:43995)
I20260618 23:29:24.320293 23778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.130@KRBTEST.COM'} at 127.8.39.130:43435
W20260618 23:29:24.394783 24107 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:29:24.395011 24107 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:29:24.395036 24107 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:29:24.396202 24107 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:29:24.396246 24107 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:29:24.396274 24107 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.131
I20260618 23:29:24.397957 24107 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.131:44023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.39.131
--webserver_port=34965
--tserver_master_addrs=127.8.39.190:37277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24107
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.131
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:29:24.398226 24107 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:29:24.398471 24107 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:29:24.403524 24112 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:29:24.403529 24113 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:29:24.403600 24115 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:29:24.404202 24107 server_base.cc:1034] running on GCE node
I20260618 23:29:24.404445 24107 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:29:24.404675 24107 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:29:24.405829 24107 hybrid_clock.cc:648] HybridClock initialized: now 1781825364405803 us; error 36 us; skew 500 ppm
Jun 18 23:29:24 dist-test-slave-47qt krb5kdc[23197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825364, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:29:24.408298 24107 init.cc:377] Logged in from keytab as kudu/127.8.39.131@KRBTEST.COM (short username kudu)
I20260618 23:29:24.408725 24107 webserver.cc:460] Webserver started at http://127.8.39.131:34965/ using document root <none> and password file <none>
I20260618 23:29:24.408977 24107 fs_manager.cc:359] Metadata directory not provided
I20260618 23:29:24.409056 24107 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:29:24.410966 24107 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:29:24.411676 24122 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:29:24.411870 24107 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:29:24.412034 24107 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/wal
uuid: "b2d1a5a2c767498a8b563af3d0df6d59"
format_stamp: "Formatted at 2026-06-18 23:29:03 on dist-test-slave-47qt"
server_key: "f6c7a8d5b17323f8d28798c1e395b3fa"
server_key_iv: "e70e159d50348c134592ea1a77010022"
server_key_version: "encryptionkey@0"
I20260618 23:29:24.412431 24107 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:29:24.425493 24107 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:29:24.428007 24107 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:29:24.428192 24107 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:29:24.428447 24107 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:29:24.428768 24107 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:29:24.428824 24107 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:29:24.428859 24107 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:29:24.428889 24107 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:29:24.435724 24107 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.131:44023
I20260618 23:29:24.435799 24235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.131:44023 every 8 connection(s)
I20260618 23:29:24.436159 24107 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
Jun 18 23:29:24 dist-test-slave-47qt krb5kdc[23197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825364, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:29:24.443924 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 24107
I20260618 23:29:24.446837 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:24.437177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:50729 (local address 127.8.39.190:37277)
0618 23:29:24.437242 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0618 23:29:24.437245 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:29:24.437547 (+ 302us) server_negotiation.cc:407] Connection header received
0618 23:29:24.438044 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:29:24.438045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:29:24.438055 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:29:24.438113 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:29:24.439351 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.439856 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.440960 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.441172 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.441228 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:24.443394 (+ 2166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:29:24.443395 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:29:24.443396 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:29:24.443402 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:29:24.445260 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:29:24.445789 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:29:24.445791 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:29:24.445791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:29:24.445848 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:29:24.446142 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:29:24.446143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:29:24.446143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:29:24.446245 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:29:24.446353 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0618 23:29:24.446626 (+ 273us) server_negotiation.cc:299] Negotiation successful
0618 23:29:24.446713 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260618 23:29:24.446899 24238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:24.437206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.131:50729)
0618 23:29:24.437417 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:24.437454 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0618 23:29:24.437943 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:29:24.438188 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:29:24.438191 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:29:24.438788 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:29:24.439181 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:24.439191 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.439981 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.439983 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:24.440863 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:24.440865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.441249 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.441250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:24.441360 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:24.441948 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:29:24.441956 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:29:24.443309 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:29:24.445345 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:29:24.445350 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:29:24.445352 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:29:24.445665 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:29:24.445937 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:29:24.445938 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:29:24.445938 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:29:24.446072 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:29:24.446347 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:29:24.446353 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:29:24.446485 (+ 132us) client_negotiation.cc:769] Sending connection context
0618 23:29:24.446578 (+ 93us) client_negotiation.cc:240] Negotiation successful
0618 23:29:24.446702 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":102,"threads_started":1}
I20260618 23:29:24.447837 24236 heartbeater.cc:347] Connected to a master server at 127.8.39.190:37277
I20260618 23:29:24.447950 24236 heartbeater.cc:464] Registering TS with master...
I20260618 23:29:24.448179 24236 heartbeater.cc:511] Master 127.8.39.190:37277 requested a full tablet report, sending...
I20260618 23:29:24.448635 23778 ts_manager.cc:194] Registered new tserver with Master: b2d1a5a2c767498a8b563af3d0df6d59 (127.8.39.131:44023)
I20260618 23:29:24.449071 23778 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.131@KRBTEST.COM'} at 127.8.39.131:50729
I20260618 23:29:24.451689 24240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:24.444311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51550 (local address 127.8.39.190:37277)
0618 23:29:24.444511 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0618 23:29:24.444513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:29:24.444565 (+ 52us) server_negotiation.cc:407] Connection header received
0618 23:29:24.444643 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:29:24.444644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:29:24.444654 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:29:24.444709 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:29:24.445521 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.446241 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.447226 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.447567 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.447643 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:24.449043 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:29:24.449044 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:29:24.449045 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:29:24.449051 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:29:24.450561 (+ 1510us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:29:24.451046 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:29:24.451046 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:29:24.451047 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:29:24.451097 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:29:24.451247 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:29:24.451248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:29:24.451248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:29:24.451345 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:29:24.451476 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0618 23:29:24.451504 (+ 28us) server_negotiation.cc:299] Negotiation successful
0618 23:29:24.451564 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":184,"thread_start_us":70,"threads_started":1}
I20260618 23:29:24.452618 8350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:29:24.455657 24240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:24.452956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51562 (local address 127.8.39.190:37277)
0618 23:29:24.453028 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0618 23:29:24.453030 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:29:24.453145 (+ 115us) server_negotiation.cc:407] Connection header received
0618 23:29:24.453221 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:29:24.453222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:29:24.453251 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:29:24.453352 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:29:24.453476 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.454060 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.454796 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.455017 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.455102 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:24.455257 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:29:24.455312 (+ 55us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0618 23:29:24.455588 (+ 276us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51562: token expired
Metrics: {"server-negotiator.queue_time_us":50}
W20260618 23:29:24.455714 24240 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51562: token expired
W20260618 23:29:24.456584 23707 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.8.39.190:37277)
I20260618 23:29:24.464017 24240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:24.457033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51576 (local address 127.8.39.190:37277)
0618 23:29:24.457094 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0618 23:29:24.457096 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:29:24.457105 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:29:24.457168 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:29:24.457169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:29:24.457181 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:29:24.457245 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:29:24.457938 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.458757 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.459601 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.459809 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.459893 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:24.461349 (+ 1456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:29:24.461351 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:29:24.461351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:29:24.461358 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:29:24.462841 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:29:24.463209 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:29:24.463210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:29:24.463211 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:29:24.463269 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:29:24.463481 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:29:24.463482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:29:24.463482 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:29:24.463624 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:29:24.463731 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0618 23:29:24.463835 (+ 104us) server_negotiation.cc:299] Negotiation successful
0618 23:29:24.463917 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260618 23:29:24.465250 23778 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51576:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:29:24.466872 23778 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:29:24.469146 23900 tablet_service.cc:1467] Processing CreateTablet for tablet ddc3559c171849359f1112fffc2202ce (DEFAULT_TABLE table=test-table [id=8fe345d22888472393899c64da1bea7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:29:24.469362 23900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddc3559c171849359f1112fffc2202ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:29:24.470886 23858 tablet_bootstrap.cc:492] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369: Bootstrap starting.
I20260618 23:29:24.471240 23858 tablet_bootstrap.cc:654] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369: Neither blocks nor log segments found. Creating new log.
I20260618 23:29:24.472141 23858 tablet_bootstrap.cc:492] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369: No bootstrap required, opened a new log
I20260618 23:29:24.472190 23858 ts_tablet_manager.cc:1397] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260618 23:29:24.472362 23858 raft_consensus.cc:348] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09091bdf5ff24353a21b85abb493dbb2" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 43995 } } peers { permanent_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44023 } } peers { permanent_uuid: "f0eff757ff2c48608da677b47879e369" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 44949 } }
I20260618 23:29:24.472429 23858 raft_consensus.cc:374] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:29:24.472458 23858 raft_consensus.cc:729] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0eff757ff2c48608da677b47879e369, State: Initialized, Role: FOLLOWER
I20260618 23:29:24.472514 23858 consensus_queue.cc:260] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09091bdf5ff24353a21b85abb493dbb2" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 43995 } } peers { permanent_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44023 } } peers { permanent_uuid: "f0eff757ff2c48608da677b47879e369" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 44949 } }
I20260618 23:29:24.472630 23858 ts_tablet_manager.cc:1428] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260618 23:29:24.474025 24243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:24.469169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39232 (local address 127.8.39.130:43995)
0618 23:29:24.469338 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0618 23:29:24.469343 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:29:24.469352 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:29:24.469415 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:29:24.469417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:29:24.469430 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:29:24.469495 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:29:24.469657 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.470561 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.471913 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.473592 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.473670 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:24.473724 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0618 23:29:24.473924 (+ 200us) server_negotiation.cc:299] Negotiation successful
0618 23:29:24.473965 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":104,"threads_started":1}
I20260618 23:29:24.474030 23973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:24.468968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:43995 (local address 127.0.0.1:39232)
0618 23:29:24.469009 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:24.469021 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:29:24.469064 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:29:24.469519 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:29:24.469520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:29:24.469522 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:29:24.469586 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:24.469587 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.470648 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.470649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:24.471829 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:24.471831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.473692 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.473693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:24.473751 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:24.473766 (+ 15us) client_negotiation.cc:769] Sending connection context
0618 23:29:24.473909 (+ 143us) client_negotiation.cc:240] Negotiation successful
0618 23:29:24.473961 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260618 23:29:24.474769 24038 tablet_service.cc:1467] Processing CreateTablet for tablet ddc3559c171849359f1112fffc2202ce (DEFAULT_TABLE table=test-table [id=8fe345d22888472393899c64da1bea7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:29:24.474995 24242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:24.468968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:44023 (local address 127.0.0.1:53500)
0618 23:29:24.469146 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:24.469156 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:29:24.469206 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:29:24.471396 (+ 2190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:29:24.471397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:29:24.471400 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:29:24.471445 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:24.471446 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.472623 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.472623 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:24.473764 (+ 1141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:24.473766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.474701 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.474702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:24.474783 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:24.474788 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:29:24.474858 (+ 70us) client_negotiation.cc:240] Negotiation successful
0618 23:29:24.474917 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":31,"thread_start_us":75,"threads_started":1}
I20260618 23:29:24.475153 24038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddc3559c171849359f1112fffc2202ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:29:24.475270 24244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:24.471006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53500 (local address 127.8.39.131:44023)
0618 23:29:24.471222 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0618 23:29:24.471226 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:29:24.471237 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:29:24.471273 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:29:24.471275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:29:24.471289 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:29:24.471372 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:29:24.471532 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.472531 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.473860 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.474628 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.474696 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:24.474731 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0618 23:29:24.475010 (+ 279us) server_negotiation.cc:299] Negotiation successful
0618 23:29:24.475183 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":119,"threads_started":1}
I20260618 23:29:24.477769 24170 tablet_service.cc:1467] Processing CreateTablet for tablet ddc3559c171849359f1112fffc2202ce (DEFAULT_TABLE table=test-table [id=8fe345d22888472393899c64da1bea7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:29:24.478116 24170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddc3559c171849359f1112fffc2202ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:29:24.478514 24246 tablet_bootstrap.cc:492] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2: Bootstrap starting.
I20260618 23:29:24.479453 24246 tablet_bootstrap.cc:654] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2: Neither blocks nor log segments found. Creating new log.
I20260618 23:29:24.480121 24246 log.cc:824] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2: Log is configured to *not* fsync() on all Append() calls
I20260618 23:29:24.480821 24248 tablet_bootstrap.cc:492] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59: Bootstrap starting.
I20260618 23:29:24.481099 24246 tablet_bootstrap.cc:492] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2: No bootstrap required, opened a new log
I20260618 23:29:24.481225 24246 ts_tablet_manager.cc:1397] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 23:29:24.481515 24248 tablet_bootstrap.cc:654] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59: Neither blocks nor log segments found. Creating new log.
I20260618 23:29:24.481997 24248 log.cc:824] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59: Log is configured to *not* fsync() on all Append() calls
I20260618 23:29:24.482618 24248 tablet_bootstrap.cc:492] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59: No bootstrap required, opened a new log
I20260618 23:29:24.482667 24248 ts_tablet_manager.cc:1397] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260618 23:29:24.482777 24246 raft_consensus.cc:348] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09091bdf5ff24353a21b85abb493dbb2" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 43995 } } peers { permanent_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44023 } } peers { permanent_uuid: "f0eff757ff2c48608da677b47879e369" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 44949 } }
I20260618 23:29:24.482935 24246 raft_consensus.cc:374] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:29:24.482980 24246 raft_consensus.cc:729] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09091bdf5ff24353a21b85abb493dbb2, State: Initialized, Role: FOLLOWER
I20260618 23:29:24.483116 24246 consensus_queue.cc:260] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09091bdf5ff24353a21b85abb493dbb2" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 43995 } } peers { permanent_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44023 } } peers { permanent_uuid: "f0eff757ff2c48608da677b47879e369" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 44949 } }
I20260618 23:29:24.483395 24246 ts_tablet_manager.cc:1428] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260618 23:29:24.483428 24104 heartbeater.cc:503] Master 127.8.39.190:37277 was elected leader, sending a full tablet report...
I20260618 23:29:24.484515 24248 raft_consensus.cc:348] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09091bdf5ff24353a21b85abb493dbb2" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 43995 } } peers { permanent_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44023 } } peers { permanent_uuid: "f0eff757ff2c48608da677b47879e369" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 44949 } }
I20260618 23:29:24.484666 24248 raft_consensus.cc:374] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:29:24.484709 24248 raft_consensus.cc:729] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d1a5a2c767498a8b563af3d0df6d59, State: Initialized, Role: FOLLOWER
I20260618 23:29:24.484812 24248 consensus_queue.cc:260] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09091bdf5ff24353a21b85abb493dbb2" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 43995 } } peers { permanent_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44023 } } peers { permanent_uuid: "f0eff757ff2c48608da677b47879e369" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 44949 } }
I20260618 23:29:24.485080 24248 ts_tablet_manager.cc:1428] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260618 23:29:24.485096 24236 heartbeater.cc:503] Master 127.8.39.190:37277 was elected leader, sending a full tablet report...
I20260618 23:29:24.543220 24250 raft_consensus.cc:482] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:29:24.543412 24250 raft_consensus.cc:504] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09091bdf5ff24353a21b85abb493dbb2" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 43995 } } peers { permanent_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44023 } } peers { permanent_uuid: "f0eff757ff2c48608da677b47879e369" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 44949 } }
I20260618 23:29:24.543902 24250 leader_election.cc:288] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b2d1a5a2c767498a8b563af3d0df6d59 (127.8.39.131:44023), f0eff757ff2c48608da677b47879e369 (127.8.39.129:44949)
I20260618 23:29:24.546804 24244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:24.544167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:43365 (local address 127.8.39.131:44023)
0618 23:29:24.544222 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0618 23:29:24.544224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:29:24.544320 (+ 96us) server_negotiation.cc:407] Connection header received
0618 23:29:24.544353 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:29:24.544354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:29:24.544364 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:29:24.544423 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:29:24.544592 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.545086 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.546070 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.546447 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.546529 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:24.546539 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0618 23:29:24.546696 (+ 157us) server_negotiation.cc:299] Negotiation successful
0618 23:29:24.546733 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260618 23:29:24.546804 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:24.544100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:44023 (local address 127.8.39.130:43365)
0618 23:29:24.544198 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:24.544209 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:29:24.544287 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:29:24.544441 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:29:24.544443 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:29:24.544445 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:29:24.544518 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:24.544520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.545231 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.545233 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:24.545971 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:24.545973 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.546537 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.546538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:24.546620 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:24.546623 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:29:24.546690 (+ 67us) client_negotiation.cc:240] Negotiation successful
0618 23:29:24.546733 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260618 23:29:24.547127 24190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc3559c171849359f1112fffc2202ce" candidate_uuid: "09091bdf5ff24353a21b85abb493dbb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" is_pre_election: true
I20260618 23:29:24.547197 24252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:24.544121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:44949 (local address 127.8.39.130:51395)
0618 23:29:24.544312 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:24.544322 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:29:24.544391 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:29:24.544583 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:29:24.544584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:29:24.544586 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:29:24.544670 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:24.544671 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.545469 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.545471 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:24.546282 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:24.546284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.546966 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.546967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:24.547041 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:24.547045 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:29:24.547102 (+ 57us) client_negotiation.cc:240] Negotiation successful
0618 23:29:24.547136 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":105,"threads_started":1}
I20260618 23:29:24.547219 23974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:24.544347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:51395 (local address 127.8.39.129:44949)
0618 23:29:24.544404 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0618 23:29:24.544406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:29:24.544416 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:29:24.544460 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:29:24.544461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:29:24.544486 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:29:24.544580 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:29:24.544761 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.545379 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.546404 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.546891 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.546970 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:24.547053 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0618 23:29:24.547108 (+ 55us) server_negotiation.cc:299] Negotiation successful
0618 23:29:24.547144 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260618 23:29:24.547291 24190 raft_consensus.cc:2454] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09091bdf5ff24353a21b85abb493dbb2 in term 0.
I20260618 23:29:24.547459 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc3559c171849359f1112fffc2202ce" candidate_uuid: "09091bdf5ff24353a21b85abb493dbb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0eff757ff2c48608da677b47879e369" is_pre_election: true
I20260618 23:29:24.547545 23994 leader_election.cc:302] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09091bdf5ff24353a21b85abb493dbb2, b2d1a5a2c767498a8b563af3d0df6d59; no voters:
I20260618 23:29:24.547626 23920 raft_consensus.cc:2454] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09091bdf5ff24353a21b85abb493dbb2 in term 0.
I20260618 23:29:24.547667 24250 raft_consensus.cc:2781] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:29:24.547744 24250 raft_consensus.cc:482] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:29:24.547784 24250 raft_consensus.cc:3037] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:29:24.548547 24250 raft_consensus.cc:504] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09091bdf5ff24353a21b85abb493dbb2" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 43995 } } peers { permanent_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44023 } } peers { permanent_uuid: "f0eff757ff2c48608da677b47879e369" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 44949 } }
I20260618 23:29:24.548723 24250 leader_election.cc:288] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2 [CANDIDATE]: Term 1 election: Requested vote from peers b2d1a5a2c767498a8b563af3d0df6d59 (127.8.39.131:44023), f0eff757ff2c48608da677b47879e369 (127.8.39.129:44949)
I20260618 23:29:24.548914 24190 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc3559c171849359f1112fffc2202ce" candidate_uuid: "09091bdf5ff24353a21b85abb493dbb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2d1a5a2c767498a8b563af3d0df6d59"
I20260618 23:29:24.548913 23920 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc3559c171849359f1112fffc2202ce" candidate_uuid: "09091bdf5ff24353a21b85abb493dbb2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f0eff757ff2c48608da677b47879e369"
I20260618 23:29:24.548990 24190 raft_consensus.cc:3037] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:29:24.549013 23920 raft_consensus.cc:3037] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:29:24.549691 24190 raft_consensus.cc:2454] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09091bdf5ff24353a21b85abb493dbb2 in term 1.
I20260618 23:29:24.549757 23920 raft_consensus.cc:2454] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09091bdf5ff24353a21b85abb493dbb2 in term 1.
I20260618 23:29:24.549960 23993 leader_election.cc:302] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09091bdf5ff24353a21b85abb493dbb2, f0eff757ff2c48608da677b47879e369; no voters:
I20260618 23:29:24.550092 24250 raft_consensus.cc:2781] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:29:24.550299 24250 raft_consensus.cc:686] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2 [term 1 LEADER]: Becoming Leader. State: Replica: 09091bdf5ff24353a21b85abb493dbb2, State: Running, Role: LEADER
I20260618 23:29:24.550385 24250 consensus_queue.cc:237] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09091bdf5ff24353a21b85abb493dbb2" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 43995 } } peers { permanent_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44023 } } peers { permanent_uuid: "f0eff757ff2c48608da677b47879e369" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 44949 } }
I20260618 23:29:24.551314 23778 catalog_manager.cc:5495] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09091bdf5ff24353a21b85abb493dbb2 (127.8.39.130). New cstate: current_term: 1 leader_uuid: "09091bdf5ff24353a21b85abb493dbb2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09091bdf5ff24353a21b85abb493dbb2" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 43995 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f0eff757ff2c48608da677b47879e369" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 44949 } health_report { overall_health: UNKNOWN } } }
W20260618 23:29:24.556556 24105 tablet.cc:2369] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:29:24.566979 24243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:24.564566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39234 (local address 127.8.39.130:43995)
0618 23:29:24.564612 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0618 23:29:24.564614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:29:24.564623 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:29:24.564679 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:29:24.564680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:29:24.564702 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:29:24.564790 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:29:24.564922 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.565468 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.566194 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:24.566411 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:24.566490 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:24.566627 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:29:24.566731 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:29:24.566826 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0618 23:29:24.566846 (+ 20us) server_negotiation.cc:299] Negotiation successful
0618 23:29:24.566867 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260618 23:29:24.569075 24190 raft_consensus.cc:1261] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59 [term 1 FOLLOWER]: Refusing update from remote peer 09091bdf5ff24353a21b85abb493dbb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:29:24.569185 23920 raft_consensus.cc:1261] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369 [term 1 FOLLOWER]: Refusing update from remote peer 09091bdf5ff24353a21b85abb493dbb2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:29:24.569475 24250 consensus_queue.cc:1035] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f0eff757ff2c48608da677b47879e369" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 44949 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:29:24.569758 24253 consensus_queue.cc:1035] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:29:24.574909 24256 mvcc.cc:205] Tried to move back new op lower bound from 7298356693270958080 to 7298356693199134720. Current Snapshot: MvccSnapshot[applied={T|T < 7298356693270958080}]
I20260618 23:29:24.577075 23778 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:29:24.577226 23778 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51576:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:29:24.578024 23778 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f: Sending DeleteTablet for 3 replicas of tablet ddc3559c171849359f1112fffc2202ce
I20260618 23:29:24.578303 24038 tablet_service.cc:1514] Processing DeleteTablet for tablet ddc3559c171849359f1112fffc2202ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:29:24 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:39232
I20260618 23:29:24.578343 24170 tablet_service.cc:1514] Processing DeleteTablet for tablet ddc3559c171849359f1112fffc2202ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:29:24 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:53500
I20260618 23:29:24.578436 23900 tablet_service.cc:1514] Processing DeleteTablet for tablet ddc3559c171849359f1112fffc2202ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:29:24 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:48904
I20260618 23:29:24.578540 23975 tablet_replica.cc:330] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369: stopping tablet replica
I20260618 23:29:24.578583 23975 raft_consensus.cc:2229] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:29:24.578577 24265 tablet_replica.cc:330] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2: stopping tablet replica
I20260618 23:29:24.578637 23975 raft_consensus.cc:2258] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:29:24.578642 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 23816
I20260618 23:29:24.578670 24265 raft_consensus.cc:2229] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2 [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:29:24.579687 24265 raft_consensus.cc:2258] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:29:24.579699 24266 tablet_replica.cc:330] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59: stopping tablet replica
I20260618 23:29:24.579806 24266 raft_consensus.cc:2229] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:29:24.579912 24266 raft_consensus.cc:2258] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:29:24.580310 24266 ts_tablet_manager.cc:1905] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:29:24.582116 24266 ts_tablet_manager.cc:1918] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:29:24.582195 24266 log.cc:1196] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/wal/wals/ddc3559c171849359f1112fffc2202ce
I20260618 23:29:24.582481 24266 ts_tablet_manager.cc:1939] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59: Deleting consensus metadata
I20260618 23:29:24.582948 23765 catalog_manager.cc:4841] TS b2d1a5a2c767498a8b563af3d0df6d59 (127.8.39.131:44023): tablet ddc3559c171849359f1112fffc2202ce (table test-table [id=8fe345d22888472393899c64da1bea7a]) successfully deleted
W20260618 23:29:24.583844 23764 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.39.129:44949 (error 108)
W20260618 23:29:24.584694 24242 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:29:24.584192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:44949 (local address 127.0.0.1:48916)
0618 23:29:24.584257 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:24.584271 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:29:24.584322 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:29:24.584653 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.129:44949: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":42}
I20260618 23:29:24.584879 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 23970
W20260618 23:29:24.584849 23764 catalog_manager.cc:4584] TS f0eff757ff2c48608da677b47879e369 (127.8.39.129:44949): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ddc3559c171849359f1112fffc2202ce: Network error: Client connection negotiation failed: client connection to 127.8.39.129:44949: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260618 23:29:24.590839 24242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:24.590663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:43995 (local address 127.0.0.1:39244)
0618 23:29:24.590743 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:24.590795 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.130:43995: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260618 23:29:24.591037 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 24107
I20260618 23:29:24.596508 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 23745
I20260618 23:29:44.602854 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:37277
--webserver_interface=127.8.39.190
--webserver_port=40913
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:37277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:29:44.686673 24268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:29:44.686887 24268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:29:44.686911 24268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:29:44.688396 24268 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260618 23:29:44.688450 24268 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260618 23:29:44.688465 24268 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260618 23:29:44.688488 24268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:29:44.688500 24268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:29:44.688513 24268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:29:44.688525 24268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:29:44.688537 24268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:29:44.690269 24268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:37277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:37277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=40913
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:29:44.690488 24268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:29:44.690661 24268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:29:44.695827 24274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:29:44.695886 24273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:29:44.695829 24268 server_base.cc:1034] running on GCE node
W20260618 23:29:44.695986 24276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:29:44.696252 24268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:29:44.696564 24268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:29:44.697714 24268 hybrid_clock.cc:648] HybridClock initialized: now 1781825384697695 us; error 41 us; skew 500 ppm
Jun 18 23:29:44 dist-test-slave-47qt krb5kdc[23197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825384, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:29:44.700412 24268 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:29:44.700888 24268 webserver.cc:460] Webserver started at http://127.8.39.190:40913/ using document root <none> and password file <none>
I20260618 23:29:44.701192 24268 fs_manager.cc:359] Metadata directory not provided
I20260618 23:29:44.701259 24268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:29:44.702622 24268 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:29:44.703274 24283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:29:44.703464 24268 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260618 23:29:44.703567 24268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "28d09a3f812648c487d8fb8164065f7f"
format_stamp: "Formatted at 2026-06-18 23:29:03 on dist-test-slave-47qt"
server_key: "06db7f3f59455f9cbcd91986c295560a"
server_key_iv: "39715f061ebe9dc819ce3b157cfd1ba9"
server_key_version: "encryptionkey@0"
I20260618 23:29:44.703938 24268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:29:44.711237 24268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:29:44.713495 24268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:29:44.713647 24268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:29:44.719651 24268 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:37277
I20260618 23:29:44.719681 24335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:37277 every 8 connection(s)
I20260618 23:29:44.720082 24268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:29:44.721662 24336 sys_catalog.cc:263] Verifying existing consensus state
I20260618 23:29:44.722325 24336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f: Bootstrap starting.
I20260618 23:29:44.728231 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 24268
I20260618 23:29:44.728552 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.129:44949
--local_ip_for_outbound_sockets=127.8.39.129
--tserver_master_addrs=127.8.39.190:37277
--webserver_port=44159
--webserver_interface=127.8.39.129
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:29:44.731197 24336 log.cc:824] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f: Log is configured to *not* fsync() on all Append() calls
I20260618 23:29:44.733629 24336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 23:29:44.733930 24336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f: Bootstrap complete.
I20260618 23:29:44.736148 24336 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d09a3f812648c487d8fb8164065f7f" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 37277 } }
I20260618 23:29:44.736399 24336 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28d09a3f812648c487d8fb8164065f7f, State: Initialized, Role: FOLLOWER
I20260618 23:29:44.736596 24336 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d09a3f812648c487d8fb8164065f7f" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 37277 } }
I20260618 23:29:44.736686 24336 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:29:44.736739 24336 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:29:44.736788 24336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [term 2 FOLLOWER]: Advancing to term 3
I20260618 23:29:44.738246 24336 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d09a3f812648c487d8fb8164065f7f" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 37277 } }
I20260618 23:29:44.738538 24336 leader_election.cc:302] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28d09a3f812648c487d8fb8164065f7f; no voters:
I20260618 23:29:44.738745 24336 leader_election.cc:288] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [CANDIDATE]: Term 3 election: Requested vote from peers
I20260618 23:29:44.738811 24340 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [term 3 FOLLOWER]: Leader election won for term 3
I20260618 23:29:44.739169 24340 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [term 3 LEADER]: Becoming Leader. State: Replica: 28d09a3f812648c487d8fb8164065f7f, State: Running, Role: LEADER
I20260618 23:29:44.739264 24336 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:29:44.739275 24340 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d09a3f812648c487d8fb8164065f7f" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 37277 } }
I20260618 23:29:44.739699 24340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28d09a3f812648c487d8fb8164065f7f. Latest consensus state: current_term: 3 leader_uuid: "28d09a3f812648c487d8fb8164065f7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d09a3f812648c487d8fb8164065f7f" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 37277 } } }
I20260618 23:29:44.739800 24340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [sys.catalog]: This master's current role is: LEADER
I20260618 23:29:44.739699 24341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "28d09a3f812648c487d8fb8164065f7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28d09a3f812648c487d8fb8164065f7f" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 37277 } } }
I20260618 23:29:44.740088 24341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f [sys.catalog]: This master's current role is: LEADER
I20260618 23:29:44.740377 24346 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:29:44.741314 24346 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:29:44.741446 24346 catalog_manager.cc:1259] Loaded cluster ID: 9cc7d5021b2f4aed9925d525df5fda1c
I20260618 23:29:44.741686 24346 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:29:44.742534 24346 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:29:44.742713 24346 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f: Loaded TSK: 0
I20260618 23:29:44.742753 24346 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 28d09a3f812648c487d8fb8164065f7f: Loaded TSK: 1
F20260618 23:29:44.742969 24346 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f2bba5329ad google::LogMessage::Fail() at ??:0
@ 0x7f2bba53588a google::LogMessage::SendToLog() at ??:0
@ 0x7f2bba5324de google::LogMessage::Flush() at ??:0
@ 0x7f2bba533a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f2bbcc9f104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f2bbcc9cd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f2bc248c1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f2bc248c48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f2bc248c594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f2bc2452c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f2bc245cad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f2bbc4000d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f2bbc3f5954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f2bbd2596db start_thread at ??:0
@ 0x7f2bb903061f clone at ??:0
W20260618 23:29:44.833397 24338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:29:44.833628 24338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:29:44.833666 24338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:29:44.834810 24338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:29:44.834877 24338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:29:44.834928 24338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.129
I20260618 23:29:44.836401 24338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.129:44949
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.39.129
--webserver_port=44159
--tserver_master_addrs=127.8.39.190:37277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.129
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:29:44.836655 24338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:29:44.836880 24338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:29:44.842541 24377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:29:44.842674 24378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:29:44.842563 24380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:29:44.842898 24338 server_base.cc:1034] running on GCE node
I20260618 23:29:44.843138 24338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:29:44.843372 24338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:29:44.844545 24338 hybrid_clock.cc:648] HybridClock initialized: now 1781825384844531 us; error 29 us; skew 500 ppm
Jun 18 23:29:44 dist-test-slave-47qt krb5kdc[23197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825384, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:29:44.847048 24338 init.cc:377] Logged in from keytab as kudu/127.8.39.129@KRBTEST.COM (short username kudu)
I20260618 23:29:44.847525 24338 webserver.cc:460] Webserver started at http://127.8.39.129:44159/ using document root <none> and password file <none>
I20260618 23:29:44.847744 24338 fs_manager.cc:359] Metadata directory not provided
I20260618 23:29:44.847808 24338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:29:44.849231 24338 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:29:44.849936 24387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:29:44.850140 24338 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:29:44.850243 24338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/wal
uuid: "f0eff757ff2c48608da677b47879e369"
format_stamp: "Formatted at 2026-06-18 23:29:03 on dist-test-slave-47qt"
server_key: "1acb970c37529997eee9ea3ba9447865"
server_key_iv: "aaf8994287dcaa88f275596a9d1ac3e0"
server_key_version: "encryptionkey@0"
I20260618 23:29:44.850606 24338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:29:44.859674 24338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:29:44.862324 24338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:29:44.862515 24338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:29:44.862761 24338 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:29:44.863247 24394 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260618 23:29:44.865427 24338 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260618 23:29:44.865502 24338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260618 23:29:44.865548 24338 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260618 23:29:44.866185 24338 ts_tablet_manager.cc:610] Registered 1 tablets
I20260618 23:29:44.866241 24338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260618 23:29:44.866297 24394 tablet_bootstrap.cc:492] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369: Bootstrap starting.
I20260618 23:29:44.873647 24338 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.129:44949
I20260618 23:29:44.873684 24501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.129:44949 every 8 connection(s)
I20260618 23:29:44.874096 24338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
I20260618 23:29:44.878285 24394 log.cc:824] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369: Log is configured to *not* fsync() on all Append() calls
I20260618 23:29:44.882244 24394 tablet_bootstrap.cc:492] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 23:29:44.882613 24394 tablet_bootstrap.cc:492] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369: Bootstrap complete.
I20260618 23:29:44.883857 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 24338
I20260618 23:29:44.883872 24394 ts_tablet_manager.cc:1397] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.011s
I20260618 23:29:44.884176 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.130:43995
--local_ip_for_outbound_sockets=127.8.39.130
--tserver_master_addrs=127.8.39.190:37277
--webserver_port=42311
--webserver_interface=127.8.39.130
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:29:44.885509 24394 raft_consensus.cc:348] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09091bdf5ff24353a21b85abb493dbb2" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 43995 } } peers { permanent_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44023 } } peers { permanent_uuid: "f0eff757ff2c48608da677b47879e369" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 44949 } }
I20260618 23:29:44.885896 24394 raft_consensus.cc:729] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0eff757ff2c48608da677b47879e369, State: Initialized, Role: FOLLOWER
I20260618 23:29:44.886051 24394 consensus_queue.cc:260] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09091bdf5ff24353a21b85abb493dbb2" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 43995 } } peers { permanent_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44023 } } peers { permanent_uuid: "f0eff757ff2c48608da677b47879e369" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 44949 } }
I20260618 23:29:44.886344 24394 ts_tablet_manager.cc:1428] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
W20260618 23:29:44.955296 24504 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:29:44.874780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.129:56477)
0618 23:29:44.874951 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:44.874994 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0618 23:29:44.875470 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:29:44.955080 (+ 79610us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":97,"threads_started":1}
W20260618 23:29:44.955581 24502 heartbeater.cc:644] Failed to heartbeat to 127.8.39.190:37277 (0 consecutive failures): Network error: Failed to ping master at 127.8.39.190:37277: Client connection negotiation failed: client connection to 127.8.39.190:37277: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260618 23:29:44.955953 24504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:44.955815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.129:56297)
0618 23:29:44.955876 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:44.955920 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260618 23:29:44.956475 24504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:44.956365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.129:32823)
0618 23:29:44.956421 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:44.956451 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260618 23:29:44.956650 24502 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260618 23:29:44.971977 24506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:29:44.972236 24506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:29:44.972280 24506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:29:44.973572 24506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:29:44.973632 24506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:29:44.973680 24506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.130
I20260618 23:29:44.975205 24506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.130:43995
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.39.130
--webserver_port=42311
--tserver_master_addrs=127.8.39.190:37277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.130
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:29:44.975489 24506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:29:44.975724 24506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:29:44.980793 24512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:29:44.980814 24513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:29:44.980803 24515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:29:44.980994 24506 server_base.cc:1034] running on GCE node
I20260618 23:29:44.981271 24506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:29:44.981539 24506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:29:44.982693 24506 hybrid_clock.cc:648] HybridClock initialized: now 1781825384982682 us; error 35 us; skew 500 ppm
Jun 18 23:29:44 dist-test-slave-47qt krb5kdc[23197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825384, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:29:44.985092 24506 init.cc:377] Logged in from keytab as kudu/127.8.39.130@KRBTEST.COM (short username kudu)
I20260618 23:29:44.985581 24506 webserver.cc:460] Webserver started at http://127.8.39.130:42311/ using document root <none> and password file <none>
I20260618 23:29:44.985777 24506 fs_manager.cc:359] Metadata directory not provided
I20260618 23:29:44.985818 24506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:29:44.987089 24506 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:29:44.987759 24522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:29:44.987967 24506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:29:44.988058 24506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/wal
uuid: "09091bdf5ff24353a21b85abb493dbb2"
format_stamp: "Formatted at 2026-06-18 23:29:03 on dist-test-slave-47qt"
server_key: "556924fd87ce3b01649349b02518e3dc"
server_key_iv: "e4ec62d0aa341e2513a59ed118b87ca6"
server_key_version: "encryptionkey@0"
I20260618 23:29:44.988404 24506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:29:45.004021 24506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:29:45.006594 24506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:29:45.006776 24506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:29:45.007040 24506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:29:45.007524 24529 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260618 23:29:45.009011 24506 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260618 23:29:45.009101 24506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260618 23:29:45.009171 24506 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260618 23:29:45.009882 24506 ts_tablet_manager.cc:610] Registered 1 tablets
I20260618 23:29:45.009938 24506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260618 23:29:45.010118 24529 tablet_bootstrap.cc:492] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2: Bootstrap starting.
I20260618 23:29:45.016705 24506 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.130:43995
I20260618 23:29:45.016793 24636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.130:43995 every 8 connection(s)
I20260618 23:29:45.017094 24506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
I20260618 23:29:45.018163 24639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:45.017773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.130:52729)
0618 23:29:45.017961 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:45.018033 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":79,"threads_started":1}
W20260618 23:29:45.018354 24637 heartbeater.cc:644] Failed to heartbeat to 127.8.39.190:37277 (0 consecutive failures): Network error: Failed to ping master at 127.8.39.190:37277: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
I20260618 23:29:45.018659 24639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:45.018561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.130:35905)
0618 23:29:45.018607 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:45.018632 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260618 23:29:45.018915 24639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:45.018838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.130:40833)
0618 23:29:45.018876 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:45.018897 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260618 23:29:45.019050 24637 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260618 23:29:45.019791 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 24506
I20260618 23:29:45.020102 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.131:44023
--local_ip_for_outbound_sockets=127.8.39.131
--tserver_master_addrs=127.8.39.190:37277
--webserver_port=34965
--webserver_interface=127.8.39.131
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:29:45.021857 24529 log.cc:824] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2: Log is configured to *not* fsync() on all Append() calls
I20260618 23:29:45.023923 24529 tablet_bootstrap.cc:492] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 23:29:45.024197 24529 tablet_bootstrap.cc:492] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2: Bootstrap complete.
I20260618 23:29:45.025224 24529 ts_tablet_manager.cc:1397] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.015s
I20260618 23:29:45.026420 24529 raft_consensus.cc:348] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09091bdf5ff24353a21b85abb493dbb2" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 43995 } } peers { permanent_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44023 } } peers { permanent_uuid: "f0eff757ff2c48608da677b47879e369" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 44949 } }
I20260618 23:29:45.026687 24529 raft_consensus.cc:729] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09091bdf5ff24353a21b85abb493dbb2, State: Initialized, Role: FOLLOWER
I20260618 23:29:45.026866 24529 consensus_queue.cc:260] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09091bdf5ff24353a21b85abb493dbb2" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 43995 } } peers { permanent_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44023 } } peers { permanent_uuid: "f0eff757ff2c48608da677b47879e369" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 44949 } }
I20260618 23:29:45.027125 24529 ts_tablet_manager.cc:1428] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260618 23:29:45.027501 24639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:45.027397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.130:44433)
0618 23:29:45.027448 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:45.027481 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260618 23:29:45.027797 24639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:45.027715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.130:52085)
0618 23:29:45.027760 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:45.027782 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260618 23:29:45.105463 24640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:29:45.105686 24640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:29:45.105712 24640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:29:45.106959 24640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:29:45.107008 24640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:29:45.107035 24640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.131
I20260618 23:29:45.108915 24640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.131:44023
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.39.131
--webserver_port=34965
--tserver_master_addrs=127.8.39.190:37277
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.131
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:29:45.109264 24640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:29:45.109548 24640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:29:45.115072 24648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:29:45.115072 24650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:29:45.115090 24647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:29:45.115425 24640 server_base.cc:1034] running on GCE node
I20260618 23:29:45.115682 24640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:29:45.115921 24640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:29:45.117065 24640 hybrid_clock.cc:648] HybridClock initialized: now 1781825385117044 us; error 39 us; skew 500 ppm
Jun 18 23:29:45 dist-test-slave-47qt krb5kdc[23197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825385, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:29:45.119866 24640 init.cc:377] Logged in from keytab as kudu/127.8.39.131@KRBTEST.COM (short username kudu)
I20260618 23:29:45.120414 24640 webserver.cc:460] Webserver started at http://127.8.39.131:34965/ using document root <none> and password file <none>
I20260618 23:29:45.120795 24640 fs_manager.cc:359] Metadata directory not provided
I20260618 23:29:45.121006 24640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:29:45.123097 24640 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:29:45.123837 24657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:29:45.124099 24640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:29:45.124181 24640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/wal
uuid: "b2d1a5a2c767498a8b563af3d0df6d59"
format_stamp: "Formatted at 2026-06-18 23:29:03 on dist-test-slave-47qt"
server_key: "f6c7a8d5b17323f8d28798c1e395b3fa"
server_key_iv: "e70e159d50348c134592ea1a77010022"
server_key_version: "encryptionkey@0"
I20260618 23:29:45.124580 24640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:29:45.145848 24640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:29:45.148233 24640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:29:45.148376 24640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:29:45.148651 24640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:29:45.148983 24640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:29:45.149036 24640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:29:45.149075 24640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:29:45.149101 24640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:29:45.155437 24640 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.131:44023
I20260618 23:29:45.155479 24770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.131:44023 every 8 connection(s)
I20260618 23:29:45.155815 24640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
I20260618 23:29:45.156778 24773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:45.156426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.131:55869)
0618 23:29:45.156565 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:45.156634 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
W20260618 23:29:45.157259 24771 heartbeater.cc:644] Failed to heartbeat to 127.8.39.190:37277 (0 consecutive failures): Network error: Failed to ping master at 127.8.39.190:37277: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
I20260618 23:29:45.157317 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 24640
I20260618 23:29:45.157544 24773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:45.157457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.131:40945)
0618 23:29:45.157498 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:45.157526 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260618 23:29:45.157831 24773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:45.157721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.131:58613)
0618 23:29:45.157791 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:45.157817 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260618 23:29:45.157938 24771 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260618 23:29:45.957515 24775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:45.957023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.129:42031)
0618 23:29:45.957359 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:45.957426 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":60,"threads_started":1}
I20260618 23:29:46.028676 24776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:46.028360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.130:56983)
0618 23:29:46.028551 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:46.028601 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":74,"threads_started":1}
I20260618 23:29:46.158649 24777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:46.158420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.131:53015)
0618 23:29:46.158557 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:46.158594 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260618 23:29:46.294384 24778 raft_consensus.cc:482] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:29:46.294560 24778 raft_consensus.cc:504] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09091bdf5ff24353a21b85abb493dbb2" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 43995 } } peers { permanent_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44023 } } peers { permanent_uuid: "f0eff757ff2c48608da677b47879e369" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 44949 } }
I20260618 23:29:46.295109 24778 leader_election.cc:288] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 09091bdf5ff24353a21b85abb493dbb2 (127.8.39.130:43995), b2d1a5a2c767498a8b563af3d0df6d59 (127.8.39.131:44023)
Jun 18 23:29:46 dist-test-slave-47qt krb5kdc[23197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825384, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for kudu/127.8.39.130@KRBTEST.COM
Jun 18 23:29:46 dist-test-slave-47qt krb5kdc[23197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825384, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for kudu/127.8.39.131@KRBTEST.COM
I20260618 23:29:46.311332 24775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:46.295249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:44023 (local address 127.8.39.129:47155)
0618 23:29:46.295304 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:46.295316 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:29:46.295380 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:29:46.296195 (+ 815us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:29:46.296196 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:29:46.296413 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:29:46.296502 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:46.296507 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:46.297648 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:46.297649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:46.298431 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:46.298433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:46.299266 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:46.299267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:46.299326 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:46.305367 (+ 6041us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:29:46.305372 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:29:46.307485 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:29:46.309649 (+ 2164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:29:46.309650 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:29:46.309650 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:29:46.309945 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:29:46.310308 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:29:46.310309 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:29:46.310309 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:29:46.310426 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:29:46.310874 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:29:46.310875 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:29:46.310913 (+ 38us) client_negotiation.cc:769] Sending connection context
0618 23:29:46.310983 (+ 70us) client_negotiation.cc:240] Negotiation successful
0618 23:29:46.311217 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":5}
I20260618 23:29:46.311352 24780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:46.295372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:47155 (local address 127.8.39.131:44023)
0618 23:29:46.295500 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0618 23:29:46.295512 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 23:29:46.295547 (+ 35us) server_negotiation.cc:407] Connection header received
0618 23:29:46.296034 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:29:46.296036 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:29:46.296070 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:29:46.296213 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:29:46.296639 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:46.297544 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:46.298547 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:46.299122 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:46.299298 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:46.307618 (+ 8320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:29:46.307621 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:29:46.307621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:29:46.307640 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:29:46.309558 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:29:46.310086 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:29:46.310089 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:29:46.310089 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:29:46.310183 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:29:46.310512 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:29:46.310513 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:29:46.310513 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:29:46.310795 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:29:46.310906 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0618 23:29:46.311065 (+ 159us) server_negotiation.cc:299] Negotiation successful
0618 23:29:46.311212 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":66,"threads_started":1}
I20260618 23:29:46.311357 24781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:46.295504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:33431 (local address 127.8.39.130:43995)
0618 23:29:46.295661 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0618 23:29:46.295667 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 23:29:46.295683 (+ 16us) server_negotiation.cc:407] Connection header received
0618 23:29:46.295766 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:29:46.295768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:29:46.295816 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:29:46.295957 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:29:46.296630 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:46.297547 (+ 917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:46.298540 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:46.298934 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:46.299071 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:46.307324 (+ 8253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:29:46.307330 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:29:46.307330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:29:46.307344 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:29:46.309361 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:29:46.309873 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:29:46.309877 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:29:46.309877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:29:46.309992 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:29:46.310308 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:29:46.310309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:29:46.310309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:29:46.310638 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:29:46.310702 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0618 23:29:46.311111 (+ 409us) server_negotiation.cc:299] Negotiation successful
0618 23:29:46.311238 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":56,"threads_started":1}
I20260618 23:29:46.311443 24779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:46.295250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:43995 (local address 127.8.39.129:33431)
0618 23:29:46.295383 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:46.295393 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:29:46.295445 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:29:46.296016 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:29:46.296019 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:29:46.296328 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:29:46.296502 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:46.296507 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:46.297663 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:46.297664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:46.298432 (+ 768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:46.298435 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:46.299044 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:46.299045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:46.299154 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:46.305337 (+ 6183us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:29:46.305348 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:29:46.307113 (+ 1765us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:29:46.309454 (+ 2341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:29:46.309461 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:29:46.309464 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:29:46.309789 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:29:46.310078 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:29:46.310079 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:29:46.310079 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:29:46.310215 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:29:46.310720 (+ 505us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:29:46.310726 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:29:46.310880 (+ 154us) client_negotiation.cc:769] Sending connection context
0618 23:29:46.311017 (+ 137us) client_negotiation.cc:240] Negotiation successful
0618 23:29:46.311304 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":230,"thread_start_us":67,"threads_started":1}
I20260618 23:29:46.312557 24591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc3559c171849359f1112fffc2202ce" candidate_uuid: "f0eff757ff2c48608da677b47879e369" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "09091bdf5ff24353a21b85abb493dbb2" is_pre_election: true
I20260618 23:29:46.312690 24591 raft_consensus.cc:2454] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f0eff757ff2c48608da677b47879e369 in term 1.
W20260618 23:29:46.312949 24391 leader_election.cc:341] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b2d1a5a2c767498a8b563af3d0df6d59 (127.8.39.131:44023): Not found: Tablet not found: ddc3559c171849359f1112fffc2202ce
I20260618 23:29:46.313025 24391 leader_election.cc:302] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09091bdf5ff24353a21b85abb493dbb2, f0eff757ff2c48608da677b47879e369; no voters: b2d1a5a2c767498a8b563af3d0df6d59
I20260618 23:29:46.313148 24778 raft_consensus.cc:2781] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260618 23:29:46.313211 24778 raft_consensus.cc:482] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:29:46.313241 24778 raft_consensus.cc:3037] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369 [term 1 FOLLOWER]: Advancing to term 2
I20260618 23:29:46.314316 24778 raft_consensus.cc:504] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09091bdf5ff24353a21b85abb493dbb2" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 43995 } } peers { permanent_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44023 } } peers { permanent_uuid: "f0eff757ff2c48608da677b47879e369" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 44949 } }
I20260618 23:29:46.314491 24778 leader_election.cc:288] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369 [CANDIDATE]: Term 2 election: Requested vote from peers 09091bdf5ff24353a21b85abb493dbb2 (127.8.39.130:43995), b2d1a5a2c767498a8b563af3d0df6d59 (127.8.39.131:44023)
I20260618 23:29:46.314680 24591 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ddc3559c171849359f1112fffc2202ce" candidate_uuid: "f0eff757ff2c48608da677b47879e369" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "09091bdf5ff24353a21b85abb493dbb2"
I20260618 23:29:46.314774 24591 raft_consensus.cc:3037] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2 [term 1 FOLLOWER]: Advancing to term 2
W20260618 23:29:46.314821 24391 leader_election.cc:341] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer b2d1a5a2c767498a8b563af3d0df6d59 (127.8.39.131:44023): Not found: Tablet not found: ddc3559c171849359f1112fffc2202ce
I20260618 23:29:46.316622 24591 raft_consensus.cc:2454] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f0eff757ff2c48608da677b47879e369 in term 2.
I20260618 23:29:46.316890 24389 leader_election.cc:302] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 09091bdf5ff24353a21b85abb493dbb2, f0eff757ff2c48608da677b47879e369; no voters: b2d1a5a2c767498a8b563af3d0df6d59
I20260618 23:29:46.317351 24778 raft_consensus.cc:2781] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369 [term 2 FOLLOWER]: Leader election won for term 2
I20260618 23:29:46.317531 24778 raft_consensus.cc:686] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369 [term 2 LEADER]: Becoming Leader. State: Replica: f0eff757ff2c48608da677b47879e369, State: Running, Role: LEADER
I20260618 23:29:46.317649 24778 consensus_queue.cc:237] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09091bdf5ff24353a21b85abb493dbb2" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 43995 } } peers { permanent_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44023 } } peers { permanent_uuid: "f0eff757ff2c48608da677b47879e369" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 44949 } }
I20260618 23:29:46.317857 24779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:46.317742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.129:58631)
0618 23:29:46.317787 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:46.317822 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260618 23:29:46.817353 24591 raft_consensus.cc:1261] T ddc3559c171849359f1112fffc2202ce P 09091bdf5ff24353a21b85abb493dbb2 [term 2 FOLLOWER]: Refusing update from remote peer f0eff757ff2c48608da677b47879e369: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260618 23:29:46.817812 24778 consensus_queue.cc:1035] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09091bdf5ff24353a21b85abb493dbb2" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 43995 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260618 23:29:46.819104 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:46.818931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.130:47235)
0618 23:29:46.819013 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:46.819059 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61,"thread_start_us":41,"threads_started":1}
W20260618 23:29:46.819988 24391 consensus_peers.cc:487] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369 -> Peer b2d1a5a2c767498a8b563af3d0df6d59 (127.8.39.131:44023): Couldn't send request to peer b2d1a5a2c767498a8b563af3d0df6d59. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ddc3559c171849359f1112fffc2202ce. This is attempt 1: this message will repeat every 5th retry.
I20260618 23:29:46.820107 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:46.820003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.130:34299)
0618 23:29:46.820055 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:46.820082 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260618 23:29:46.820143 24790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:46.819877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.129:33295)
0618 23:29:46.820013 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:46.820047 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":50,"threads_started":1}
I20260618 23:29:47.159884 24791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:47.159520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.131:53879)
0618 23:29:47.159757 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:47.159813 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":90,"threads_started":1}
I20260618 23:29:47.230113 24792 ts_tablet_manager.cc:927] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59: Initiating tablet copy from peer f0eff757ff2c48608da677b47879e369 (127.8.39.129:44949)
I20260618 23:29:47.230384 24792 tablet_copy_client.cc:316] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59: tablet copy: Beginning tablet copy session from remote peer at address 127.8.39.129:44949
Jun 18 23:29:47 dist-test-slave-47qt krb5kdc[23197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825385, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for kudu/127.8.39.129@KRBTEST.COM
I20260618 23:29:47.239502 24794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:47.230832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:32797 (local address 127.8.39.129:44949)
0618 23:29:47.231021 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0618 23:29:47.231026 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:29:47.231034 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:29:47.231058 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:29:47.231059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:29:47.231070 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:29:47.231141 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:29:47.231655 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:47.232352 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:47.233193 (+ 841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:29:47.233513 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:29:47.233582 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:47.236563 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:29:47.236580 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:29:47.236581 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:29:47.236590 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:29:47.238282 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:29:47.238685 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:29:47.238687 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:29:47.238688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:29:47.238743 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:29:47.238963 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:29:47.238964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:29:47.238964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:29:47.239136 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:29:47.239233 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0618 23:29:47.239350 (+ 117us) server_negotiation.cc:299] Negotiation successful
0618 23:29:47.239425 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":101,"threads_started":1}
I20260618 23:29:47.239511 24791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:47.230677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:44949 (local address 127.8.39.131:32797)
0618 23:29:47.230746 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:47.230764 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 23:29:47.230829 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:29:47.231191 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:29:47.231197 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:29:47.231495 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:29:47.231567 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:47.231569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:47.232465 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:47.232466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:47.233054 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:29:47.233056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:29:47.233688 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:29:47.233690 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:29:47.233808 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:29:47.234606 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:29:47.234612 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:29:47.236421 (+ 1809us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:29:47.238378 (+ 1957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:29:47.238389 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:29:47.238391 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:29:47.238601 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:29:47.238811 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:29:47.238812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:29:47.238812 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:29:47.238866 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:29:47.239208 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:29:47.239214 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:29:47.239262 (+ 48us) client_negotiation.cc:769] Sending connection context
0618 23:29:47.239327 (+ 65us) client_negotiation.cc:240] Negotiation successful
0618 23:29:47.239424 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260618 23:29:47.240031 24476 tablet_copy_service.cc:140] P f0eff757ff2c48608da677b47879e369: Received BeginTabletCopySession request for tablet ddc3559c171849359f1112fffc2202ce from peer b2d1a5a2c767498a8b563af3d0df6d59 ({username='kudu', principal='kudu/127.8.39.131@KRBTEST.COM'} at 127.8.39.131:32797)
I20260618 23:29:47.240094 24476 tablet_copy_service.cc:161] P f0eff757ff2c48608da677b47879e369: Beginning new tablet copy session on tablet ddc3559c171849359f1112fffc2202ce from peer b2d1a5a2c767498a8b563af3d0df6d59 at {username='kudu', principal='kudu/127.8.39.131@KRBTEST.COM'} at 127.8.39.131:32797: session id = b2d1a5a2c767498a8b563af3d0df6d59-ddc3559c171849359f1112fffc2202ce
I20260618 23:29:47.240530 24476 tablet_copy_source_session.cc:205] T ddc3559c171849359f1112fffc2202ce P f0eff757ff2c48608da677b47879e369: Tablet Copy: opened 0 blocks and 1 log segments
I20260618 23:29:47.241544 24792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ddc3559c171849359f1112fffc2202ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:29:47.244232 24792 tablet_copy_client.cc:791] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59: tablet copy: Starting download of 0 data blocks...
I20260618 23:29:47.244405 24792 tablet_copy_client.cc:655] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59: tablet copy: Starting download of 1 WAL segments...
I20260618 23:29:47.245437 24792 tablet_copy_client.cc:531] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260618 23:29:47.246215 24792 tablet_bootstrap.cc:492] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59: Bootstrap starting.
I20260618 23:29:47.247603 24792 log.cc:824] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59: Log is configured to *not* fsync() on all Append() calls
I20260618 23:29:47.250092 24792 tablet_bootstrap.cc:492] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260618 23:29:47.250336 24792 tablet_bootstrap.cc:492] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59: Bootstrap complete.
I20260618 23:29:47.250476 24792 ts_tablet_manager.cc:1397] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260618 23:29:47.251509 24792 raft_consensus.cc:348] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09091bdf5ff24353a21b85abb493dbb2" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 43995 } } peers { permanent_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44023 } } peers { permanent_uuid: "f0eff757ff2c48608da677b47879e369" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 44949 } }
I20260618 23:29:47.251644 24792 raft_consensus.cc:729] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d1a5a2c767498a8b563af3d0df6d59, State: Initialized, Role: FOLLOWER
I20260618 23:29:47.251777 24792 consensus_queue.cc:260] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "09091bdf5ff24353a21b85abb493dbb2" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 43995 } } peers { permanent_uuid: "b2d1a5a2c767498a8b563af3d0df6d59" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 44023 } } peers { permanent_uuid: "f0eff757ff2c48608da677b47879e369" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 44949 } }
I20260618 23:29:47.252050 24792 ts_tablet_manager.cc:1428] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260618 23:29:47.252352 24476 tablet_copy_service.cc:342] P f0eff757ff2c48608da677b47879e369: Request end of tablet copy session b2d1a5a2c767498a8b563af3d0df6d59-ddc3559c171849359f1112fffc2202ce received from {username='kudu', principal='kudu/127.8.39.131@KRBTEST.COM'} at 127.8.39.131:32797
I20260618 23:29:47.252404 24476 tablet_copy_service.cc:433] P f0eff757ff2c48608da677b47879e369: ending tablet copy session b2d1a5a2c767498a8b563af3d0df6d59-ddc3559c171849359f1112fffc2202ce on tablet ddc3559c171849359f1112fffc2202ce with peer b2d1a5a2c767498a8b563af3d0df6d59
I20260618 23:29:47.252382 24791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:47.252246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.131:34307)
0618 23:29:47.252314 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:47.252351 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260618 23:29:47.252745 24791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:47.252625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.131:36879)
0618 23:29:47.252660 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:47.252717 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260618 23:29:47.704345 24798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:47.704131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.129:39951)
0618 23:29:47.704250 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:47.704299 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":41,"threads_started":1}
I20260618 23:29:47.704372 24725 raft_consensus.cc:1203] T ddc3559c171849359f1112fffc2202ce P b2d1a5a2c767498a8b563af3d0df6d59 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260618 23:29:47.704969 24791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:47.704769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.131:52343)
0618 23:29:47.704853 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:47.704910 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260618 23:29:47.820828 24799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:47.820549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.130:53993)
0618 23:29:47.820690 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:47.820735 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":80,"threads_started":1}
I20260618 23:29:48.705192 24800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:48.704872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.129:46987)
0618 23:29:48.705041 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:48.705078 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":67,"threads_started":1}
I20260618 23:29:48.705684 24801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:48.705473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.131:33689)
0618 23:29:48.705601 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:48.705639 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260618 23:29:48.822186 24802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:48.821610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.130:33941)
0618 23:29:48.822065 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:48.822135 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":379,"threads_started":1}
I20260618 23:29:49.706278 24803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:49.705837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.129:54219)
0618 23:29:49.706167 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:49.706217 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":100,"threads_started":1}
I20260618 23:29:49.706490 24804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:49.706263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.131:50901)
0618 23:29:49.706399 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:49.706437 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260618 23:29:49.823282 24805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:49.822983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.130:50225)
0618 23:29:49.823188 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:49.823229 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":109,"threads_started":1}
I20260618 23:29:50.707333 24806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:50.707070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.129:53087)
0618 23:29:50.707224 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:50.707269 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":62,"threads_started":1}
I20260618 23:29:50.707420 24807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:50.707181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.131:44097)
0618 23:29:50.707345 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:50.707382 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":68,"threads_started":1}
I20260618 23:29:50.707803 24806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:50.707716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.129:34147)
0618 23:29:50.707761 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:50.707785 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260618 23:29:50.824186 24808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:50.823907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.130:33053)
0618 23:29:50.824085 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:50.824135 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":63,"threads_started":1}
I20260618 23:29:51.708453 24809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:51.707979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.131:43701)
0618 23:29:51.708339 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:51.708387 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":335,"thread_start_us":112,"threads_started":1}
I20260618 23:29:51.708627 24810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:51.708390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.129:48967)
0618 23:29:51.708509 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:51.708571 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":60,"threads_started":1}
I20260618 23:29:51.825079 24811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:51.824804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.130:54063)
0618 23:29:51.824980 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:51.825033 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":78,"threads_started":1}
I20260618 23:29:52.709367 24813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:52.709150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.129:48645)
0618 23:29:52.709277 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:52.709310 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":57,"threads_started":1}
I20260618 23:29:52.709360 24812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:52.709095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.131:40899)
0618 23:29:52.709274 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:52.709315 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":106,"threads_started":1}
I20260618 23:29:52.825940 24814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:52.825691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.130:49805)
0618 23:29:52.825849 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:52.825894 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":70,"threads_started":1}
I20260618 23:29:53.710003 24815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:53.709806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.129:46639)
0618 23:29:53.709927 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:53.709955 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260618 23:29:53.710126 24816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:53.709825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.131:49575)
0618 23:29:53.710013 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:53.710055 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":143,"threads_started":1}
I20260618 23:29:53.826797 24817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:53.826547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.130:49733)
0618 23:29:53.826676 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:53.826718 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":85,"threads_started":1}
I20260618 23:29:54.710757 24819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:54.710567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.131:57241)
0618 23:29:54.710668 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:54.710709 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":45,"threads_started":1}
I20260618 23:29:54.710790 24818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:54.710512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.129:47755)
0618 23:29:54.710712 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:54.710749 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":80,"threads_started":1}
I20260618 23:29:54.827620 24820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:54.827397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.130:45317)
0618 23:29:54.827532 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:54.827574 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":58,"threads_started":1}
I20260618 23:29:55.711506 24821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:55.711285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.131:50447)
0618 23:29:55.711417 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:55.711459 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":59,"threads_started":1}
I20260618 23:29:55.711513 24822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:55.711303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.129:51843)
0618 23:29:55.711440 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:55.711473 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":77,"threads_started":1}
I20260618 23:29:55.828521 24823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:55.828307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.130:48367)
0618 23:29:55.828437 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:55.828477 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":59,"threads_started":1}
I20260618 23:29:56.712450 24824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:56.712167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.131:42713)
0618 23:29:56.712336 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:56.712389 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":66,"threads_started":1}
I20260618 23:29:56.712450 24825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:56.712186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.129:39723)
0618 23:29:56.712366 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:56.712404 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":101,"threads_started":1}
I20260618 23:29:56.829387 24826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:56.829086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.130:55055)
0618 23:29:56.829299 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:56.829341 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":143,"threads_started":1}
I20260618 23:29:57.713258 24828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:57.713000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.131:34145)
0618 23:29:57.713192 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:57.713223 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":92,"threads_started":1}
I20260618 23:29:57.713258 24827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:57.713001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.129:47167)
0618 23:29:57.713162 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:57.713200 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":90,"threads_started":1}
I20260618 23:29:57.830345 24829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:57.830113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.130:50469)
0618 23:29:57.830241 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:57.830283 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260618 23:29:58.714155 24831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:58.713777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.129:37567)
0618 23:29:58.714078 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:58.714113 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":230,"threads_started":1}
I20260618 23:29:58.714202 24830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:58.713762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.131:46971)
0618 23:29:58.714001 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:58.714045 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":57,"threads_started":1}
I20260618 23:29:58.831126 24832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:58.830877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.130:39215)
0618 23:29:58.831009 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:58.831047 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260618 23:29:59.714956 24833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:59.714726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.131:38059)
0618 23:29:59.714885 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:59.714914 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":74,"threads_started":1}
I20260618 23:29:59.714956 24834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:59.714726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.129:58855)
0618 23:29:59.714873 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:59.714910 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":71,"threads_started":1}
I20260618 23:29:59.831843 24835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:29:59.831572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37277 (local address 127.8.39.130:58353)
0618 23:29:59.831752 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0618 23:29:59.831794 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.190:37277: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":66,"threads_started":1}
I20260618 23:30:00.166222 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 24338
I20260618 23:30:00.171473 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 24506
I20260618 23:30:00.175984 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 24640
2026-06-18T23:30:00Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58307 ms)
[----------] 30 tests from SecurityITest (346642 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:30:00 dist-test-slave-47qt krb5kdc[24838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:30:00 dist-test-slave-47qt krb5kdc[24838](info): set up 2 sockets
Jun 18 23:30:00 dist-test-slave-47qt krb5kdc[24838](info): commencing operation
krb5kdc: starting...
W20260618 23:30:01.265198 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:30:01 dist-test-slave-47qt krb5kdc[24838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:30:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:30:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.39.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
I20260618 23:30:01.388479 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:32931
--webserver_interface=127.8.39.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:45121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:32931
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:30:01.469583 24851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:01.469815 24851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:01.469853 24851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:01.470999 24851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:30:01.471071 24851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:01.471095 24851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:01.471117 24851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:30:01.471145 24851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:30:01.472630 24851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:45121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:32931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:32931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:01.472922 24851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:01.473181 24851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:01.478333 24856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:01.478296 24857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:01.478361 24859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:01.478329 24851 server_base.cc:1034] running on GCE node
I20260618 23:30:01.478783 24851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:01.479072 24851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:01.480238 24851 hybrid_clock.cc:648] HybridClock initialized: now 1781825401480195 us; error 58 us; skew 500 ppm
Jun 18 23:30:01 dist-test-slave-47qt krb5kdc[24838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825401, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:01.482954 24851 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:30:01.483464 24851 webserver.cc:460] Webserver started at http://127.8.39.190:41345/ using document root <none> and password file <none>
I20260618 23:30:01.483697 24851 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:01.483765 24851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:01.483922 24851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:01.484983 24851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "d8fc87a9d1754eb4b361f2c1a6141e68"
format_stamp: "Formatted at 2026-06-18 23:30:01 on dist-test-slave-47qt"
server_key: "617c3cfd6fac38b11dd2762b6bc16ca0"
server_key_iv: "039fb93bfecbe780a7bff901a78dce30"
server_key_version: "encryptionkey@0"
I20260618 23:30:01.485368 24851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "d8fc87a9d1754eb4b361f2c1a6141e68"
format_stamp: "Formatted at 2026-06-18 23:30:01 on dist-test-slave-47qt"
server_key: "617c3cfd6fac38b11dd2762b6bc16ca0"
server_key_iv: "039fb93bfecbe780a7bff901a78dce30"
server_key_version: "encryptionkey@0"
I20260618 23:30:01.486707 24851 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:30:01.487532 24866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:01.487766 24851 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:30:01.487879 24851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "d8fc87a9d1754eb4b361f2c1a6141e68"
format_stamp: "Formatted at 2026-06-18 23:30:01 on dist-test-slave-47qt"
server_key: "617c3cfd6fac38b11dd2762b6bc16ca0"
server_key_iv: "039fb93bfecbe780a7bff901a78dce30"
server_key_version: "encryptionkey@0"
I20260618 23:30:01.487968 24851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:01.529493 24851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:01.531613 24851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:01.531797 24851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:01.537422 24851 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:32931
I20260618 23:30:01.537456 24918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:32931 every 8 connection(s)
I20260618 23:30:01.537845 24851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:30:01.538769 24919 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:01.540992 24919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8fc87a9d1754eb4b361f2c1a6141e68: Bootstrap starting.
I20260618 23:30:01.541605 24919 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8fc87a9d1754eb4b361f2c1a6141e68: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:01.541898 24919 log.cc:824] T 00000000000000000000000000000000 P d8fc87a9d1754eb4b361f2c1a6141e68: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:01.542887 24919 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8fc87a9d1754eb4b361f2c1a6141e68: No bootstrap required, opened a new log
I20260618 23:30:01.544252 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 24851
I20260618 23:30:01.544377 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:30:01.544586 8350 external_mini_cluster.cc:1428] Setting key 4b5616d74586129b37f85c0141eb468a
I20260618 23:30:01.545882 24919 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8fc87a9d1754eb4b361f2c1a6141e68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8fc87a9d1754eb4b361f2c1a6141e68" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 32931 } }
I20260618 23:30:01.546075 24919 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8fc87a9d1754eb4b361f2c1a6141e68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:01.546119 24919 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8fc87a9d1754eb4b361f2c1a6141e68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8fc87a9d1754eb4b361f2c1a6141e68, State: Initialized, Role: FOLLOWER
I20260618 23:30:01.546280 24919 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8fc87a9d1754eb4b361f2c1a6141e68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8fc87a9d1754eb4b361f2c1a6141e68" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 32931 } }
I20260618 23:30:01.546377 24919 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8fc87a9d1754eb4b361f2c1a6141e68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:30:01.546413 24919 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8fc87a9d1754eb4b361f2c1a6141e68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:30:01.546473 24919 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8fc87a9d1754eb4b361f2c1a6141e68 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:01.547590 24919 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8fc87a9d1754eb4b361f2c1a6141e68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8fc87a9d1754eb4b361f2c1a6141e68" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 32931 } }
I20260618 23:30:01.547739 24919 leader_election.cc:302] T 00000000000000000000000000000000 P d8fc87a9d1754eb4b361f2c1a6141e68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8fc87a9d1754eb4b361f2c1a6141e68; no voters:
I20260618 23:30:01.547915 24919 leader_election.cc:288] T 00000000000000000000000000000000 P d8fc87a9d1754eb4b361f2c1a6141e68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:30:01.547961 24924 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8fc87a9d1754eb4b361f2c1a6141e68 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:30:01.548149 24924 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8fc87a9d1754eb4b361f2c1a6141e68 [term 1 LEADER]: Becoming Leader. State: Replica: d8fc87a9d1754eb4b361f2c1a6141e68, State: Running, Role: LEADER
I20260618 23:30:01.548189 24919 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8fc87a9d1754eb4b361f2c1a6141e68 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:30:01.548306 24924 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8fc87a9d1754eb4b361f2c1a6141e68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8fc87a9d1754eb4b361f2c1a6141e68" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 32931 } }
I20260618 23:30:01.548748 24925 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8fc87a9d1754eb4b361f2c1a6141e68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8fc87a9d1754eb4b361f2c1a6141e68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8fc87a9d1754eb4b361f2c1a6141e68" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 32931 } } }
I20260618 23:30:01.548820 24925 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8fc87a9d1754eb4b361f2c1a6141e68 [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:01.549055 24926 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8fc87a9d1754eb4b361f2c1a6141e68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8fc87a9d1754eb4b361f2c1a6141e68. Latest consensus state: current_term: 1 leader_uuid: "d8fc87a9d1754eb4b361f2c1a6141e68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8fc87a9d1754eb4b361f2c1a6141e68" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 32931 } } }
I20260618 23:30:01.549194 24926 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8fc87a9d1754eb4b361f2c1a6141e68 [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:01.549667 24931 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:30:01.550271 24931 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 18 23:30:01 dist-test-slave-47qt krb5kdc[24838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825401, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:30:01.551971 24931 catalog_manager.cc:1347] Generated new cluster ID: a52d2ab50691404b95a82abbc371c698
I20260618 23:30:01.552018 24931 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:30:01.555749 24923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:01.545244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50758 (local address 127.8.39.190:32931)
0618 23:30:01.545409 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:01.545417 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:01.545451 (+ 34us) server_negotiation.cc:407] Connection header received
0618 23:30:01.545850 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:01.545856 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:01.545904 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:01.546035 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:01.546898 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:01.547875 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:01.549014 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:01.549674 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:01.550003 (+ 329us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:01.552437 (+ 2434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:01.552441 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:01.552441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:01.552453 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:01.554112 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:01.554477 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:01.554480 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:01.554481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:01.554587 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:01.554797 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:01.554798 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:01.554798 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:01.555039 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:01.555212 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:01.555278 (+ 66us) server_negotiation.cc:299] Negotiation successful
0618 23:30:01.555522 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":82,"threads_started":1}
I20260618 23:30:01.559233 24931 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:30:01.559861 24931 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:30:01.566028 24931 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8fc87a9d1754eb4b361f2c1a6141e68: Generated new TSK 0
I20260618 23:30:01.566228 24931 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.39.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
I20260618 23:30:01.608189 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.129:0
--local_ip_for_outbound_sockets=127.8.39.129
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:32931
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:45121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:30:01.692499 24945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:01.692709 24945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:01.692735 24945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:01.694000 24945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:01.694044 24945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:01.694072 24945 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.129
I20260618 23:30:01.696347 24945 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:45121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:32931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.129
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:01.696653 24945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:01.696909 24945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:01.703594 24951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:01.703601 24953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:01.703601 24950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:01.704002 24945 server_base.cc:1034] running on GCE node
I20260618 23:30:01.704227 24945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:01.704521 24945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:01.705684 24945 hybrid_clock.cc:648] HybridClock initialized: now 1781825401705665 us; error 34 us; skew 500 ppm
Jun 18 23:30:01 dist-test-slave-47qt krb5kdc[24838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825401, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:01.708544 24945 init.cc:377] Logged in from keytab as kudu/127.8.39.129@KRBTEST.COM (short username kudu)
I20260618 23:30:01.709029 24945 webserver.cc:460] Webserver started at http://127.8.39.129:45001/ using document root <none> and password file <none>
I20260618 23:30:01.709281 24945 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:01.709352 24945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:01.709504 24945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:01.710593 24945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-0/data/instance:
uuid: "8d0ef75fb58d475bac3e176af7ca9e1a"
format_stamp: "Formatted at 2026-06-18 23:30:01 on dist-test-slave-47qt"
server_key: "0c206542d5f07110f06889f63b9eb11c"
server_key_iv: "946573e2d38f48a01d13ce9a49c539a2"
server_key_version: "encryptionkey@0"
I20260618 23:30:01.710956 24945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance:
uuid: "8d0ef75fb58d475bac3e176af7ca9e1a"
format_stamp: "Formatted at 2026-06-18 23:30:01 on dist-test-slave-47qt"
server_key: "0c206542d5f07110f06889f63b9eb11c"
server_key_iv: "946573e2d38f48a01d13ce9a49c539a2"
server_key_version: "encryptionkey@0"
I20260618 23:30:01.712333 24945 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:30:01.713215 24960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:01.713418 24945 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:30:01.713529 24945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-0/wal
uuid: "8d0ef75fb58d475bac3e176af7ca9e1a"
format_stamp: "Formatted at 2026-06-18 23:30:01 on dist-test-slave-47qt"
server_key: "0c206542d5f07110f06889f63b9eb11c"
server_key_iv: "946573e2d38f48a01d13ce9a49c539a2"
server_key_version: "encryptionkey@0"
I20260618 23:30:01.713616 24945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:01.726140 24945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:01.728960 24945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:01.729213 24945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:01.729490 24945 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:30:01.729833 24945 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:30:01.729887 24945 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:01.729921 24945 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:30:01.729950 24945 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:01.738000 24945 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.129:43227
I20260618 23:30:01.738044 25073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.129:43227 every 8 connection(s)
I20260618 23:30:01.738519 24945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
I20260618 23:30:01.743816 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 24945
I20260618 23:30:01.743902 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance
I20260618 23:30:01.744139 8350 external_mini_cluster.cc:1428] Setting key 260a4f68ffda5b3ada42a3dc11b49b36
Jun 18 23:30:01 dist-test-slave-47qt krb5kdc[24838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825401, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:30:01.749541 24923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:01.739387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:56693 (local address 127.8.39.190:32931)
0618 23:30:01.739454 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:01.739456 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:01.739654 (+ 198us) server_negotiation.cc:407] Connection header received
0618 23:30:01.740138 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:01.740139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:01.740152 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:01.740218 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:01.741178 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:01.741679 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:01.742625 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:01.742837 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:01.742887 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:01.745545 (+ 2658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:01.745548 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:01.745548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:01.745598 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:01.747623 (+ 2025us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:01.748203 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:01.748204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:01.748205 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:01.748311 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:01.748640 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:01.748641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:01.748641 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:01.748757 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:01.748952 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:01.749326 (+ 374us) server_negotiation.cc:299] Negotiation successful
0618 23:30:01.749436 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260618 23:30:01.749764 25076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:01.739363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:32931 (local address 127.8.39.129:56693)
0618 23:30:01.739527 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:01.739569 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:01.740058 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:01.740270 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:01.740273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:01.740852 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:01.741048 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:01.741053 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:01.741761 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:01.741762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:01.742554 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:01.742555 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:01.742909 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:01.742910 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:01.743003 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:01.743532 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:30:01.743539 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:01.745362 (+ 1823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:01.747713 (+ 2351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:01.747721 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:01.747724 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:01.748052 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:01.748367 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:01.748368 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:01.748368 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:01.748530 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:01.748997 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:01.749003 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:01.749179 (+ 176us) client_negotiation.cc:769] Sending connection context
0618 23:30:01.749304 (+ 125us) client_negotiation.cc:240] Negotiation successful
0618 23:30:01.749524 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
I20260618 23:30:01.750617 25074 heartbeater.cc:347] Connected to a master server at 127.8.39.190:32931
I20260618 23:30:01.750821 25074 heartbeater.cc:464] Registering TS with master...
I20260618 23:30:01.751034 25074 heartbeater.cc:511] Master 127.8.39.190:32931 requested a full tablet report, sending...
I20260618 23:30:01.751628 24882 ts_manager.cc:194] Registered new tserver with Master: 8d0ef75fb58d475bac3e176af7ca9e1a (127.8.39.129:43227)
I20260618 23:30:01.752638 24882 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.129@KRBTEST.COM'} at 127.8.39.129:56693
WARNING: no policy specified for kudu/127.8.39.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
I20260618 23:30:01.772451 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.130:0
--local_ip_for_outbound_sockets=127.8.39.130
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:32931
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:45121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:30:01.856369 25079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:01.856585 25079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:01.856611 25079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:01.857812 25079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:01.857892 25079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:01.857941 25079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.130
I20260618 23:30:01.859479 25079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:45121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:32931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.130
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:01.859733 25079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:01.859978 25079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 23:30:01.865209 25079 server_base.cc:1034] running on GCE node
W20260618 23:30:01.865190 25087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:01.865175 25085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:01.865183 25084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:01.865816 25079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:01.866088 25079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:01.867266 25079 hybrid_clock.cc:648] HybridClock initialized: now 1781825401867260 us; error 41 us; skew 500 ppm
Jun 18 23:30:01 dist-test-slave-47qt krb5kdc[24838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825401, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:01.870396 25079 init.cc:377] Logged in from keytab as kudu/127.8.39.130@KRBTEST.COM (short username kudu)
I20260618 23:30:01.870954 25079 webserver.cc:460] Webserver started at http://127.8.39.130:41477/ using document root <none> and password file <none>
I20260618 23:30:01.871210 25079 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:01.871281 25079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:01.871459 25079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:01.872525 25079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-1/data/instance:
uuid: "72cbb234812b43e5b7f2043e37298801"
format_stamp: "Formatted at 2026-06-18 23:30:01 on dist-test-slave-47qt"
server_key: "8eb6b79264cb96d709da582e73675af6"
server_key_iv: "e5dc4968829a7c1576c46ae22843ba66"
server_key_version: "encryptionkey@0"
I20260618 23:30:01.872888 25079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance:
uuid: "72cbb234812b43e5b7f2043e37298801"
format_stamp: "Formatted at 2026-06-18 23:30:01 on dist-test-slave-47qt"
server_key: "8eb6b79264cb96d709da582e73675af6"
server_key_iv: "e5dc4968829a7c1576c46ae22843ba66"
server_key_version: "encryptionkey@0"
I20260618 23:30:01.874440 25079 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:30:01.875802 25094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:01.876057 25079 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:30:01.876180 25079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-1/wal
uuid: "72cbb234812b43e5b7f2043e37298801"
format_stamp: "Formatted at 2026-06-18 23:30:01 on dist-test-slave-47qt"
server_key: "8eb6b79264cb96d709da582e73675af6"
server_key_iv: "e5dc4968829a7c1576c46ae22843ba66"
server_key_version: "encryptionkey@0"
I20260618 23:30:01.876276 25079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:01.908740 25079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:01.911975 25079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:01.912178 25079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:01.912467 25079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:30:01.912786 25079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:30:01.912842 25079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:01.912885 25079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:30:01.912910 25079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:01.919272 25079 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.130:40055
I20260618 23:30:01.919308 25207 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.130:40055 every 8 connection(s)
I20260618 23:30:01.919833 25079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
Jun 18 23:30:01 dist-test-slave-47qt krb5kdc[24838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825401, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:30:01.928087 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 25079
I20260618 23:30:01.928207 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance
I20260618 23:30:01.928512 8350 external_mini_cluster.cc:1428] Setting key a49c9db84ee1bcfd23f07204594d70dc
I20260618 23:30:01.930495 24923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:01.921459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:45835 (local address 127.8.39.190:32931)
0618 23:30:01.921569 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:01.921572 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:01.921884 (+ 312us) server_negotiation.cc:407] Connection header received
0618 23:30:01.922380 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:01.922382 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:01.922400 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:01.922482 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:01.923506 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:01.924017 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:01.925015 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:01.925223 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:01.925282 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:01.927304 (+ 2022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:01.927305 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:01.927306 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:01.927312 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:01.929057 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:01.929503 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:01.929505 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:01.929505 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:01.929570 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:01.929798 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:01.929799 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:01.929799 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:01.929948 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:01.930029 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:01.930256 (+ 227us) server_negotiation.cc:299] Negotiation successful
0618 23:30:01.930352 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260618 23:30:01.930512 25210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:01.921499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:32931 (local address 127.8.39.130:45835)
0618 23:30:01.921681 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:01.921735 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:01.922226 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:01.922567 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:01.922571 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:01.923155 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:01.923359 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:01.923363 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:01.924111 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:01.924112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:01.924904 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:01.924906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:01.925298 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:01.925299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:01.925398 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:01.925862 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:30:01.925869 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:01.927222 (+ 1353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:01.929167 (+ 1945us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:01.929174 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:01.929177 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:01.929407 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:01.929632 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:01.929633 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:01.929633 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:01.929726 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:01.930017 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:01.930021 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:01.930134 (+ 113us) client_negotiation.cc:769] Sending connection context
0618 23:30:01.930214 (+ 80us) client_negotiation.cc:240] Negotiation successful
0618 23:30:01.930331 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":92,"threads_started":1}
I20260618 23:30:01.930970 25208 heartbeater.cc:347] Connected to a master server at 127.8.39.190:32931
I20260618 23:30:01.931080 25208 heartbeater.cc:464] Registering TS with master...
I20260618 23:30:01.931275 25208 heartbeater.cc:511] Master 127.8.39.190:32931 requested a full tablet report, sending...
I20260618 23:30:01.931710 24882 ts_manager.cc:194] Registered new tserver with Master: 72cbb234812b43e5b7f2043e37298801 (127.8.39.130:40055)
I20260618 23:30:01.932076 24882 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.130@KRBTEST.COM'} at 127.8.39.130:45835
WARNING: no policy specified for kudu/127.8.39.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
I20260618 23:30:01.956216 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.131:0
--local_ip_for_outbound_sockets=127.8.39.131
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:32931
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:45121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:30:02.050102 25213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:02.050338 25213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:02.050379 25213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:02.052047 25213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:02.052145 25213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:02.052199 25213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.131
I20260618 23:30:02.053858 25213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:45121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:32931
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.131
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:02.054139 25213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:02.054407 25213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:02.061169 25221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:02.061178 25218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:02.061240 25219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:02.061554 25213 server_base.cc:1034] running on GCE node
I20260618 23:30:02.061775 25213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:02.062064 25213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:02.063365 25213 hybrid_clock.cc:648] HybridClock initialized: now 1781825402063286 us; error 87 us; skew 500 ppm
Jun 18 23:30:02 dist-test-slave-47qt krb5kdc[24838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825402, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:02.067476 25213 init.cc:377] Logged in from keytab as kudu/127.8.39.131@KRBTEST.COM (short username kudu)
I20260618 23:30:02.067984 25213 webserver.cc:460] Webserver started at http://127.8.39.131:41085/ using document root <none> and password file <none>
I20260618 23:30:02.068253 25213 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:02.068331 25213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:02.068487 25213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:02.069643 25213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-2/data/instance:
uuid: "ed7fab8d0ff24280b05e656e36526b07"
format_stamp: "Formatted at 2026-06-18 23:30:02 on dist-test-slave-47qt"
server_key: "c84b48b298a4dbf91f87feff43856d80"
server_key_iv: "d3bf850e6ebf93c52ea5acff18ad0a3c"
server_key_version: "encryptionkey@0"
I20260618 23:30:02.070032 25213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance:
uuid: "ed7fab8d0ff24280b05e656e36526b07"
format_stamp: "Formatted at 2026-06-18 23:30:02 on dist-test-slave-47qt"
server_key: "c84b48b298a4dbf91f87feff43856d80"
server_key_iv: "d3bf850e6ebf93c52ea5acff18ad0a3c"
server_key_version: "encryptionkey@0"
I20260618 23:30:02.071592 25213 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:30:02.072546 25228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:02.072908 25213 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:30:02.073028 25213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-2/wal
uuid: "ed7fab8d0ff24280b05e656e36526b07"
format_stamp: "Formatted at 2026-06-18 23:30:02 on dist-test-slave-47qt"
server_key: "c84b48b298a4dbf91f87feff43856d80"
server_key_iv: "d3bf850e6ebf93c52ea5acff18ad0a3c"
server_key_version: "encryptionkey@0"
I20260618 23:30:02.073104 25213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:02.088819 25213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:02.091886 25213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:02.092087 25213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:02.092470 25213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:30:02.092828 25213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:30:02.092890 25213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:02.092933 25213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:30:02.092968 25213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:02.100433 25213 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.131:45111
I20260618 23:30:02.100476 25341 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.131:45111 every 8 connection(s)
I20260618 23:30:02.100798 25213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
I20260618 23:30:02.101759 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 25213
I20260618 23:30:02.101850 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance
I20260618 23:30:02.102036 8350 external_mini_cluster.cc:1428] Setting key e2616298b28ef1d335add4d569af47aa
Jun 18 23:30:02 dist-test-slave-47qt krb5kdc[24838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825402, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:30:02.114328 24923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:02.102520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:42541 (local address 127.8.39.190:32931)
0618 23:30:02.102672 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:02.102685 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:02.103024 (+ 339us) server_negotiation.cc:407] Connection header received
0618 23:30:02.103579 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:02.103581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:02.103596 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:02.103687 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:02.104904 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.105679 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.107138 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.107367 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.107439 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:02.110537 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:02.110538 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:02.110538 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:02.110546 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:02.112228 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:02.112753 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:02.112757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:02.112757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:02.112866 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:02.113406 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:02.113407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:02.113407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:02.113606 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:02.113707 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:02.114112 (+ 405us) server_negotiation.cc:299] Negotiation successful
0618 23:30:02.114217 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260618 23:30:02.114640 25344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:02.102656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:32931 (local address 127.8.39.131:42541)
0618 23:30:02.102871 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:02.102921 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:02.103427 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:02.103800 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:02.103804 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:02.104421 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:02.104626 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:02.104630 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.105803 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.105804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:02.107033 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:02.107036 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.107536 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.107538 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:02.107730 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:02.108709 (+ 979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:30:02.108727 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:02.110392 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:02.112358 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:02.112366 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:02.112368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:02.112609 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:02.113007 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:02.113008 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:02.113008 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:02.113251 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:02.113725 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:02.113732 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:02.113924 (+ 192us) client_negotiation.cc:769] Sending connection context
0618 23:30:02.114046 (+ 122us) client_negotiation.cc:240] Negotiation successful
0618 23:30:02.114262 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":107,"threads_started":1}
I20260618 23:30:02.115185 25342 heartbeater.cc:347] Connected to a master server at 127.8.39.190:32931
I20260618 23:30:02.115350 25342 heartbeater.cc:464] Registering TS with master...
I20260618 23:30:02.115577 25342 heartbeater.cc:511] Master 127.8.39.190:32931 requested a full tablet report, sending...
I20260618 23:30:02.116019 24882 ts_manager.cc:194] Registered new tserver with Master: ed7fab8d0ff24280b05e656e36526b07 (127.8.39.131:45111)
I20260618 23:30:02.116408 24882 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.131@KRBTEST.COM'} at 127.8.39.131:42541
I20260618 23:30:02.125243 8350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:30:02.133905 24923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:02.126701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50772 (local address 127.8.39.190:32931)
0618 23:30:02.126807 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:02.126816 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:02.126826 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:30:02.126884 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:02.126885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:02.126895 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:02.126944 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:02.127489 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.127985 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.128749 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.128933 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.129003 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:02.130424 (+ 1421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:02.130425 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:02.130425 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:02.130431 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:02.131889 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:02.132266 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:02.132267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:02.132267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:02.132454 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:02.132919 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:02.132920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:02.132920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:02.133345 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:02.133455 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:02.133649 (+ 194us) server_negotiation.cc:299] Negotiation successful
0618 23:30:02.133724 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260618 23:30:02.136226 24882 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50772:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:30:02.138805 24882 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:30:02.148118 25354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:02.142669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:45111 (local address 127.0.0.1:40684)
0618 23:30:02.142934 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:02.142987 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:02.143141 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:02.144134 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:02.144149 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:02.144153 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:30:02.144308 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:02.144310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.145592 (+ 1282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.145594 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:02.147028 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:02.147031 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.147742 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.147743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:02.147835 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:02.147851 (+ 16us) client_negotiation.cc:769] Sending connection context
0618 23:30:02.147965 (+ 114us) client_negotiation.cc:240] Negotiation successful
0618 23:30:02.148032 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":76,"spinlock_wait_cycles":1280,"thread_start_us":116,"threads_started":1}
I20260618 23:30:02.148126 25356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:02.142818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40684 (local address 127.8.39.131:45111)
0618 23:30:02.143709 (+ 891us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:02.143715 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:02.143734 (+ 19us) server_negotiation.cc:407] Connection header received
0618 23:30:02.143776 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:02.143779 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:02.143794 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:02.144129 (+ 335us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:30:02.144414 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.145427 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.147131 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.147648 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.147744 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:02.147809 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:02.147984 (+ 175us) server_negotiation.cc:299] Negotiation successful
0618 23:30:02.148027 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":837,"thread_start_us":184,"threads_started":1}
I20260618 23:30:02.149407 25353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:02.142669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:40055 (local address 127.0.0.1:32932)
0618 23:30:02.143127 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:02.143140 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:02.143226 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:02.144158 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:02.144158 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:02.144160 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:30:02.144309 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:02.144313 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.145824 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.145825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:02.147000 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:02.147003 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.149104 (+ 2101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.149105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:02.149198 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:02.149202 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:30:02.149288 (+ 86us) client_negotiation.cc:240] Negotiation successful
0618 23:30:02.149337 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"mutex_wait_us":5,"thread_start_us":102,"threads_started":1}
I20260618 23:30:02.149646 25358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:02.143295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33618 (local address 127.8.39.129:43227)
0618 23:30:02.143835 (+ 540us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:02.143841 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:02.143853 (+ 12us) server_negotiation.cc:407] Connection header received
0618 23:30:02.143911 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:02.143913 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:02.143932 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:02.144028 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:30:02.144409 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.145669 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.147845 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.149139 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.149296 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:02.149387 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:02.149521 (+ 134us) server_negotiation.cc:299] Negotiation successful
0618 23:30:02.149568 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":495,"thread_start_us":235,"threads_started":1}
I20260618 23:30:02.149603 25276 tablet_service.cc:1467] Processing CreateTablet for tablet 27d0d4e8998641a294d56dd023445fda (DEFAULT_TABLE table=test-table [id=431b55c0eafb47a2b71291fd80b196b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:30:02.149848 25355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:02.142810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32932 (local address 127.8.39.130:40055)
0618 23:30:02.143971 (+ 1161us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:02.143975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:02.143985 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:30:02.144028 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:02.144030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:02.144046 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:02.144401 (+ 355us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:30:02.144498 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.145683 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.147902 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.148968 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.149097 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:02.149179 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:02.149716 (+ 537us) server_negotiation.cc:299] Negotiation successful
0618 23:30:02.149771 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1103,"thread_start_us":208,"threads_started":1}
I20260618 23:30:02.150053 25276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27d0d4e8998641a294d56dd023445fda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:02.149624 25357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:02.142973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:43227 (local address 127.0.0.1:33618)
0618 23:30:02.143300 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:02.143311 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:02.143359 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:02.144136 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:02.144143 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:02.144146 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:30:02.144310 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:02.144313 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.145824 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.145825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:02.147743 (+ 1918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:02.147745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.149313 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.149315 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:02.149396 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:02.149401 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:30:02.149502 (+ 101us) client_negotiation.cc:240] Negotiation successful
0618 23:30:02.149550 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":306,"mutex_wait_us":1281,"thread_start_us":91,"threads_started":1}
I20260618 23:30:02.150563 25142 tablet_service.cc:1467] Processing CreateTablet for tablet 27d0d4e8998641a294d56dd023445fda (DEFAULT_TABLE table=test-table [id=431b55c0eafb47a2b71291fd80b196b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:30:02.150777 25008 tablet_service.cc:1467] Processing CreateTablet for tablet 27d0d4e8998641a294d56dd023445fda (DEFAULT_TABLE table=test-table [id=431b55c0eafb47a2b71291fd80b196b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:30:02.150897 25142 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27d0d4e8998641a294d56dd023445fda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:02.151095 25008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 27d0d4e8998641a294d56dd023445fda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:02.153841 25359 tablet_bootstrap.cc:492] T 27d0d4e8998641a294d56dd023445fda P ed7fab8d0ff24280b05e656e36526b07: Bootstrap starting.
I20260618 23:30:02.154106 25360 tablet_bootstrap.cc:492] T 27d0d4e8998641a294d56dd023445fda P 8d0ef75fb58d475bac3e176af7ca9e1a: Bootstrap starting.
I20260618 23:30:02.154405 25361 tablet_bootstrap.cc:492] T 27d0d4e8998641a294d56dd023445fda P 72cbb234812b43e5b7f2043e37298801: Bootstrap starting.
I20260618 23:30:02.154644 25359 tablet_bootstrap.cc:654] T 27d0d4e8998641a294d56dd023445fda P ed7fab8d0ff24280b05e656e36526b07: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:02.154702 25360 tablet_bootstrap.cc:654] T 27d0d4e8998641a294d56dd023445fda P 8d0ef75fb58d475bac3e176af7ca9e1a: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:02.155112 25361 tablet_bootstrap.cc:654] T 27d0d4e8998641a294d56dd023445fda P 72cbb234812b43e5b7f2043e37298801: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:02.155272 25360 log.cc:824] T 27d0d4e8998641a294d56dd023445fda P 8d0ef75fb58d475bac3e176af7ca9e1a: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:02.155272 25359 log.cc:824] T 27d0d4e8998641a294d56dd023445fda P ed7fab8d0ff24280b05e656e36526b07: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:02.155759 25361 log.cc:824] T 27d0d4e8998641a294d56dd023445fda P 72cbb234812b43e5b7f2043e37298801: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:02.156042 25359 tablet_bootstrap.cc:492] T 27d0d4e8998641a294d56dd023445fda P ed7fab8d0ff24280b05e656e36526b07: No bootstrap required, opened a new log
I20260618 23:30:02.156035 25360 tablet_bootstrap.cc:492] T 27d0d4e8998641a294d56dd023445fda P 8d0ef75fb58d475bac3e176af7ca9e1a: No bootstrap required, opened a new log
I20260618 23:30:02.156111 25359 ts_tablet_manager.cc:1397] T 27d0d4e8998641a294d56dd023445fda P ed7fab8d0ff24280b05e656e36526b07: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260618 23:30:02.156111 25360 ts_tablet_manager.cc:1397] T 27d0d4e8998641a294d56dd023445fda P 8d0ef75fb58d475bac3e176af7ca9e1a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:30:02.156446 25361 tablet_bootstrap.cc:492] T 27d0d4e8998641a294d56dd023445fda P 72cbb234812b43e5b7f2043e37298801: No bootstrap required, opened a new log
I20260618 23:30:02.156527 25361 ts_tablet_manager.cc:1397] T 27d0d4e8998641a294d56dd023445fda P 72cbb234812b43e5b7f2043e37298801: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260618 23:30:02.158273 25361 raft_consensus.cc:348] T 27d0d4e8998641a294d56dd023445fda P 72cbb234812b43e5b7f2043e37298801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7fab8d0ff24280b05e656e36526b07" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 45111 } } peers { permanent_uuid: "8d0ef75fb58d475bac3e176af7ca9e1a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 43227 } } peers { permanent_uuid: "72cbb234812b43e5b7f2043e37298801" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 40055 } }
I20260618 23:30:02.158470 25361 raft_consensus.cc:374] T 27d0d4e8998641a294d56dd023445fda P 72cbb234812b43e5b7f2043e37298801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:02.158512 25361 raft_consensus.cc:729] T 27d0d4e8998641a294d56dd023445fda P 72cbb234812b43e5b7f2043e37298801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72cbb234812b43e5b7f2043e37298801, State: Initialized, Role: FOLLOWER
I20260618 23:30:02.158653 25361 consensus_queue.cc:260] T 27d0d4e8998641a294d56dd023445fda P 72cbb234812b43e5b7f2043e37298801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7fab8d0ff24280b05e656e36526b07" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 45111 } } peers { permanent_uuid: "8d0ef75fb58d475bac3e176af7ca9e1a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 43227 } } peers { permanent_uuid: "72cbb234812b43e5b7f2043e37298801" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 40055 } }
I20260618 23:30:02.158937 25361 ts_tablet_manager.cc:1428] T 27d0d4e8998641a294d56dd023445fda P 72cbb234812b43e5b7f2043e37298801: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260618 23:30:02.159089 25208 heartbeater.cc:503] Master 127.8.39.190:32931 was elected leader, sending a full tablet report...
I20260618 23:30:02.158921 25360 raft_consensus.cc:348] T 27d0d4e8998641a294d56dd023445fda P 8d0ef75fb58d475bac3e176af7ca9e1a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7fab8d0ff24280b05e656e36526b07" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 45111 } } peers { permanent_uuid: "8d0ef75fb58d475bac3e176af7ca9e1a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 43227 } } peers { permanent_uuid: "72cbb234812b43e5b7f2043e37298801" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 40055 } }
I20260618 23:30:02.159214 25360 raft_consensus.cc:374] T 27d0d4e8998641a294d56dd023445fda P 8d0ef75fb58d475bac3e176af7ca9e1a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:02.159092 25359 raft_consensus.cc:348] T 27d0d4e8998641a294d56dd023445fda P ed7fab8d0ff24280b05e656e36526b07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7fab8d0ff24280b05e656e36526b07" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 45111 } } peers { permanent_uuid: "8d0ef75fb58d475bac3e176af7ca9e1a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 43227 } } peers { permanent_uuid: "72cbb234812b43e5b7f2043e37298801" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 40055 } }
I20260618 23:30:02.159268 25359 raft_consensus.cc:374] T 27d0d4e8998641a294d56dd023445fda P ed7fab8d0ff24280b05e656e36526b07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:02.159265 25360 raft_consensus.cc:729] T 27d0d4e8998641a294d56dd023445fda P 8d0ef75fb58d475bac3e176af7ca9e1a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d0ef75fb58d475bac3e176af7ca9e1a, State: Initialized, Role: FOLLOWER
I20260618 23:30:02.159312 25359 raft_consensus.cc:729] T 27d0d4e8998641a294d56dd023445fda P ed7fab8d0ff24280b05e656e36526b07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed7fab8d0ff24280b05e656e36526b07, State: Initialized, Role: FOLLOWER
I20260618 23:30:02.159437 25360 consensus_queue.cc:260] T 27d0d4e8998641a294d56dd023445fda P 8d0ef75fb58d475bac3e176af7ca9e1a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7fab8d0ff24280b05e656e36526b07" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 45111 } } peers { permanent_uuid: "8d0ef75fb58d475bac3e176af7ca9e1a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 43227 } } peers { permanent_uuid: "72cbb234812b43e5b7f2043e37298801" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 40055 } }
I20260618 23:30:02.159821 25360 ts_tablet_manager.cc:1428] T 27d0d4e8998641a294d56dd023445fda P 8d0ef75fb58d475bac3e176af7ca9e1a: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260618 23:30:02.159837 25074 heartbeater.cc:503] Master 127.8.39.190:32931 was elected leader, sending a full tablet report...
I20260618 23:30:02.159435 25359 consensus_queue.cc:260] T 27d0d4e8998641a294d56dd023445fda P ed7fab8d0ff24280b05e656e36526b07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7fab8d0ff24280b05e656e36526b07" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 45111 } } peers { permanent_uuid: "8d0ef75fb58d475bac3e176af7ca9e1a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 43227 } } peers { permanent_uuid: "72cbb234812b43e5b7f2043e37298801" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 40055 } }
I20260618 23:30:02.160435 25359 ts_tablet_manager.cc:1428] T 27d0d4e8998641a294d56dd023445fda P ed7fab8d0ff24280b05e656e36526b07: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260618 23:30:02.160522 25342 heartbeater.cc:503] Master 127.8.39.190:32931 was elected leader, sending a full tablet report...
W20260618 23:30:02.170665 25209 tablet.cc:2369] T 27d0d4e8998641a294d56dd023445fda P 72cbb234812b43e5b7f2043e37298801: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 23:30:02.239533 25075 tablet.cc:2369] T 27d0d4e8998641a294d56dd023445fda P 8d0ef75fb58d475bac3e176af7ca9e1a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:30:02.241353 25367 raft_consensus.cc:482] T 27d0d4e8998641a294d56dd023445fda P ed7fab8d0ff24280b05e656e36526b07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:30:02.241534 25367 raft_consensus.cc:504] T 27d0d4e8998641a294d56dd023445fda P ed7fab8d0ff24280b05e656e36526b07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7fab8d0ff24280b05e656e36526b07" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 45111 } } peers { permanent_uuid: "8d0ef75fb58d475bac3e176af7ca9e1a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 43227 } } peers { permanent_uuid: "72cbb234812b43e5b7f2043e37298801" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 40055 } }
I20260618 23:30:02.242034 25367 leader_election.cc:288] T 27d0d4e8998641a294d56dd023445fda P ed7fab8d0ff24280b05e656e36526b07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d0ef75fb58d475bac3e176af7ca9e1a (127.8.39.129:43227), 72cbb234812b43e5b7f2043e37298801 (127.8.39.130:40055)
I20260618 23:30:02.246189 25358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:02.242488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:54141 (local address 127.8.39.129:43227)
0618 23:30:02.242569 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:02.242572 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:02.242723 (+ 151us) server_negotiation.cc:407] Connection header received
0618 23:30:02.242918 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:02.242919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:02.242949 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:02.243039 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:30:02.243333 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.244005 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.245210 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.245783 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.245864 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:02.245872 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:02.246095 (+ 223us) server_negotiation.cc:299] Negotiation successful
0618 23:30:02.246134 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260618 23:30:02.246201 25344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:02.242356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:43227 (local address 127.8.39.131:54141)
0618 23:30:02.242516 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:02.242538 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:02.242715 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:02.243094 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:02.243096 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:02.243099 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:30:02.243203 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:02.243206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.244120 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.244121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:02.245092 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:02.245094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.245878 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.245879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:02.245968 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:02.245972 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:30:02.246069 (+ 97us) client_negotiation.cc:240] Negotiation successful
0618 23:30:02.246113 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128}
I20260618 23:30:02.246587 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d0d4e8998641a294d56dd023445fda" candidate_uuid: "ed7fab8d0ff24280b05e656e36526b07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d0ef75fb58d475bac3e176af7ca9e1a" is_pre_election: true
I20260618 23:30:02.246780 25028 raft_consensus.cc:2454] T 27d0d4e8998641a294d56dd023445fda P 8d0ef75fb58d475bac3e176af7ca9e1a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed7fab8d0ff24280b05e656e36526b07 in term 0.
I20260618 23:30:02.246848 25368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:02.242356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:40055 (local address 127.8.39.131:39997)
0618 23:30:02.242968 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:02.242981 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:02.243076 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:02.243336 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:02.243337 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:02.243340 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:30:02.243528 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:02.243532 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.244692 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.244693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:02.245719 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:02.245721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.246576 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.246577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:02.246650 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:02.246654 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:30:02.246722 (+ 68us) client_negotiation.cc:240] Negotiation successful
0618 23:30:02.246761 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":189,"thread_start_us":184,"threads_started":1}
I20260618 23:30:02.246891 25355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:02.242480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:39997 (local address 127.8.39.130:40055)
0618 23:30:02.242567 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:02.242570 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:02.243124 (+ 554us) server_negotiation.cc:407] Connection header received
0618 23:30:02.243167 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:02.243169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:02.243188 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:02.243267 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:30:02.243672 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.244564 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.245867 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.246463 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.246567 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:02.246575 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:02.246749 (+ 174us) server_negotiation.cc:299] Negotiation successful
0618 23:30:02.246793 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260618 23:30:02.247015 25229 leader_election.cc:302] T 27d0d4e8998641a294d56dd023445fda P ed7fab8d0ff24280b05e656e36526b07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d0ef75fb58d475bac3e176af7ca9e1a, ed7fab8d0ff24280b05e656e36526b07; no voters:
I20260618 23:30:02.247164 25367 raft_consensus.cc:2781] T 27d0d4e8998641a294d56dd023445fda P ed7fab8d0ff24280b05e656e36526b07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:30:02.247246 25367 raft_consensus.cc:482] T 27d0d4e8998641a294d56dd023445fda P ed7fab8d0ff24280b05e656e36526b07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:30:02.247273 25367 raft_consensus.cc:3037] T 27d0d4e8998641a294d56dd023445fda P ed7fab8d0ff24280b05e656e36526b07 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:02.247278 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d0d4e8998641a294d56dd023445fda" candidate_uuid: "ed7fab8d0ff24280b05e656e36526b07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72cbb234812b43e5b7f2043e37298801" is_pre_election: true
I20260618 23:30:02.247431 25162 raft_consensus.cc:2454] T 27d0d4e8998641a294d56dd023445fda P 72cbb234812b43e5b7f2043e37298801 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed7fab8d0ff24280b05e656e36526b07 in term 0.
I20260618 23:30:02.248152 25367 raft_consensus.cc:504] T 27d0d4e8998641a294d56dd023445fda P ed7fab8d0ff24280b05e656e36526b07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7fab8d0ff24280b05e656e36526b07" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 45111 } } peers { permanent_uuid: "8d0ef75fb58d475bac3e176af7ca9e1a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 43227 } } peers { permanent_uuid: "72cbb234812b43e5b7f2043e37298801" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 40055 } }
I20260618 23:30:02.248337 25367 leader_election.cc:288] T 27d0d4e8998641a294d56dd023445fda P ed7fab8d0ff24280b05e656e36526b07 [CANDIDATE]: Term 1 election: Requested vote from peers 8d0ef75fb58d475bac3e176af7ca9e1a (127.8.39.129:43227), 72cbb234812b43e5b7f2043e37298801 (127.8.39.130:40055)
I20260618 23:30:02.248517 25028 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d0d4e8998641a294d56dd023445fda" candidate_uuid: "ed7fab8d0ff24280b05e656e36526b07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d0ef75fb58d475bac3e176af7ca9e1a"
I20260618 23:30:02.248615 25162 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "27d0d4e8998641a294d56dd023445fda" candidate_uuid: "ed7fab8d0ff24280b05e656e36526b07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72cbb234812b43e5b7f2043e37298801"
I20260618 23:30:02.248626 25028 raft_consensus.cc:3037] T 27d0d4e8998641a294d56dd023445fda P 8d0ef75fb58d475bac3e176af7ca9e1a [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:02.248682 25162 raft_consensus.cc:3037] T 27d0d4e8998641a294d56dd023445fda P 72cbb234812b43e5b7f2043e37298801 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:02.249670 25162 raft_consensus.cc:2454] T 27d0d4e8998641a294d56dd023445fda P 72cbb234812b43e5b7f2043e37298801 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed7fab8d0ff24280b05e656e36526b07 in term 1.
I20260618 23:30:02.249670 25028 raft_consensus.cc:2454] T 27d0d4e8998641a294d56dd023445fda P 8d0ef75fb58d475bac3e176af7ca9e1a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed7fab8d0ff24280b05e656e36526b07 in term 1.
I20260618 23:30:02.249843 25229 leader_election.cc:302] T 27d0d4e8998641a294d56dd023445fda P ed7fab8d0ff24280b05e656e36526b07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d0ef75fb58d475bac3e176af7ca9e1a, ed7fab8d0ff24280b05e656e36526b07; no voters:
I20260618 23:30:02.249964 25367 raft_consensus.cc:2781] T 27d0d4e8998641a294d56dd023445fda P ed7fab8d0ff24280b05e656e36526b07 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:30:02.250171 25367 raft_consensus.cc:686] T 27d0d4e8998641a294d56dd023445fda P ed7fab8d0ff24280b05e656e36526b07 [term 1 LEADER]: Becoming Leader. State: Replica: ed7fab8d0ff24280b05e656e36526b07, State: Running, Role: LEADER
I20260618 23:30:02.250283 25367 consensus_queue.cc:237] T 27d0d4e8998641a294d56dd023445fda P ed7fab8d0ff24280b05e656e36526b07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7fab8d0ff24280b05e656e36526b07" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 45111 } } peers { permanent_uuid: "8d0ef75fb58d475bac3e176af7ca9e1a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 43227 } } peers { permanent_uuid: "72cbb234812b43e5b7f2043e37298801" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 40055 } }
I20260618 23:30:02.251246 24883 catalog_manager.cc:5495] T 27d0d4e8998641a294d56dd023445fda P ed7fab8d0ff24280b05e656e36526b07 reported cstate change: term changed from 0 to 1, leader changed from <none> to ed7fab8d0ff24280b05e656e36526b07 (127.8.39.131). New cstate: current_term: 1 leader_uuid: "ed7fab8d0ff24280b05e656e36526b07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ed7fab8d0ff24280b05e656e36526b07" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 45111 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d0ef75fb58d475bac3e176af7ca9e1a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 43227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72cbb234812b43e5b7f2043e37298801" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 40055 } health_report { overall_health: UNKNOWN } } }
I20260618 23:30:02.268754 25356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:02.266289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40692 (local address 127.8.39.131:45111)
0618 23:30:02.266359 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:02.266361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:02.266370 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:30:02.266431 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:02.266432 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:02.266459 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:02.266548 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:30:02.266691 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.267182 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.267959 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.268158 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.268243 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:02.268419 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:30:02.268529 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:30:02.268625 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:02.268644 (+ 19us) server_negotiation.cc:299] Negotiation successful
0618 23:30:02.268664 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260618 23:30:02.270637 25028 raft_consensus.cc:1261] T 27d0d4e8998641a294d56dd023445fda P 8d0ef75fb58d475bac3e176af7ca9e1a [term 1 FOLLOWER]: Refusing update from remote peer ed7fab8d0ff24280b05e656e36526b07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:30:02.270658 25162 raft_consensus.cc:1261] T 27d0d4e8998641a294d56dd023445fda P 72cbb234812b43e5b7f2043e37298801 [term 1 FOLLOWER]: Refusing update from remote peer ed7fab8d0ff24280b05e656e36526b07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:30:02.271013 25367 consensus_queue.cc:1035] T 27d0d4e8998641a294d56dd023445fda P ed7fab8d0ff24280b05e656e36526b07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d0ef75fb58d475bac3e176af7ca9e1a" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 43227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:30:02.271176 25369 consensus_queue.cc:1035] T 27d0d4e8998641a294d56dd023445fda P ed7fab8d0ff24280b05e656e36526b07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72cbb234812b43e5b7f2043e37298801" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 40055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:30:02.273566 25374 mvcc.cc:205] Tried to move back new op lower bound from 7298356847697436672 to 7298356847617871872. Current Snapshot: MvccSnapshot[applied={T|T < 7298356847697436672}]
I20260618 23:30:02.273758 25373 mvcc.cc:205] Tried to move back new op lower bound from 7298356847697436672 to 7298356847617871872. Current Snapshot: MvccSnapshot[applied={T|T < 7298356847697436672}]
I20260618 23:30:02.274103 25372 mvcc.cc:205] Tried to move back new op lower bound from 7298356847697436672 to 7298356847617871872. Current Snapshot: MvccSnapshot[applied={T|T < 7298356847697436672}]
I20260618 23:30:02.276017 24881 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:30:02.276129 24881 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:30:02.277019 24881 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d8fc87a9d1754eb4b361f2c1a6141e68: Sending DeleteTablet for 3 replicas of tablet 27d0d4e8998641a294d56dd023445fda
I20260618 23:30:02.277390 25276 tablet_service.cc:1514] Processing DeleteTablet for tablet 27d0d4e8998641a294d56dd023445fda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:30:02 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:40684
I20260618 23:30:02.277479 25142 tablet_service.cc:1514] Processing DeleteTablet for tablet 27d0d4e8998641a294d56dd023445fda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:30:02 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:32932
I20260618 23:30:02.277467 25008 tablet_service.cc:1514] Processing DeleteTablet for tablet 27d0d4e8998641a294d56dd023445fda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:30:02 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:33618
I20260618 23:30:02.277874 25383 tablet_replica.cc:330] T 27d0d4e8998641a294d56dd023445fda P 72cbb234812b43e5b7f2043e37298801: stopping tablet replica
I20260618 23:30:02.278069 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 24945
I20260618 23:30:02.277990 25383 raft_consensus.cc:2229] T 27d0d4e8998641a294d56dd023445fda P 72cbb234812b43e5b7f2043e37298801 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:30:02.278518 25383 raft_consensus.cc:2258] T 27d0d4e8998641a294d56dd023445fda P 72cbb234812b43e5b7f2043e37298801 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:30:02.278859 25384 tablet_replica.cc:330] T 27d0d4e8998641a294d56dd023445fda P ed7fab8d0ff24280b05e656e36526b07: stopping tablet replica
I20260618 23:30:02.278970 25384 raft_consensus.cc:2229] T 27d0d4e8998641a294d56dd023445fda P ed7fab8d0ff24280b05e656e36526b07 [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:30:02.279235 25384 raft_consensus.cc:2258] T 27d0d4e8998641a294d56dd023445fda P ed7fab8d0ff24280b05e656e36526b07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:30:02.281056 25383 ts_tablet_manager.cc:1905] T 27d0d4e8998641a294d56dd023445fda P 72cbb234812b43e5b7f2043e37298801: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:30:02.283627 25383 ts_tablet_manager.cc:1918] T 27d0d4e8998641a294d56dd023445fda P 72cbb234812b43e5b7f2043e37298801: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:30:02.283711 25383 log.cc:1196] T 27d0d4e8998641a294d56dd023445fda P 72cbb234812b43e5b7f2043e37298801: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-1/wal/wals/27d0d4e8998641a294d56dd023445fda
I20260618 23:30:02.284011 25383 ts_tablet_manager.cc:1939] T 27d0d4e8998641a294d56dd023445fda P 72cbb234812b43e5b7f2043e37298801: Deleting consensus metadata
I20260618 23:30:02.284381 25384 ts_tablet_manager.cc:1905] T 27d0d4e8998641a294d56dd023445fda P ed7fab8d0ff24280b05e656e36526b07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:30:02.284693 24870 catalog_manager.cc:4841] TS 72cbb234812b43e5b7f2043e37298801 (127.8.39.130:40055): tablet 27d0d4e8998641a294d56dd023445fda (table test-table [id=431b55c0eafb47a2b71291fd80b196b5]) successfully deleted
I20260618 23:30:02.286576 25384 ts_tablet_manager.cc:1918] T 27d0d4e8998641a294d56dd023445fda P ed7fab8d0ff24280b05e656e36526b07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:30:02.286656 25384 log.cc:1196] T 27d0d4e8998641a294d56dd023445fda P ed7fab8d0ff24280b05e656e36526b07: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781825053542692-8350-0/minicluster-data/ts-2/wal/wals/27d0d4e8998641a294d56dd023445fda
I20260618 23:30:02.286913 25384 ts_tablet_manager.cc:1939] T 27d0d4e8998641a294d56dd023445fda P ed7fab8d0ff24280b05e656e36526b07: Deleting consensus metadata
I20260618 23:30:02.287413 24867 catalog_manager.cc:4841] TS ed7fab8d0ff24280b05e656e36526b07 (127.8.39.131:45111): tablet 27d0d4e8998641a294d56dd023445fda (table test-table [id=431b55c0eafb47a2b71291fd80b196b5]) successfully deleted
W20260618 23:30:02.288044 24867 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.39.129:43227 (error 108)
I20260618 23:30:02.288301 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 25079
I20260618 23:30:02.290145 25357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:02.289986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:43227 (local address 127.0.0.1:33626)
0618 23:30:02.290059 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:02.290109 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.129:43227: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260618 23:30:02.290274 24867 catalog_manager.cc:4584] TS 8d0ef75fb58d475bac3e176af7ca9e1a (127.8.39.129:43227): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 27d0d4e8998641a294d56dd023445fda: Network error: Client connection negotiation failed: client connection to 127.8.39.129:43227: connect: Connection refused (error 111)
I20260618 23:30:02.293903 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 25213
I20260618 23:30:02.298923 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 24851
2026-06-18T23:30:02Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2124 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-18T23:30:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:30:02Z Disabled control of system clock
I20260618 23:30:02.329955 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:42539
--webserver_interface=127.8.39.190
--webserver_port=0
--builtin_ntp_servers=127.8.39.148:44489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:42539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 23:30:02.420433 25389 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:02.420661 25389 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:02.420684 25389 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:02.422050 25389 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:30:02.422104 25389 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:02.422118 25389 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:02.422132 25389 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:30:02.422144 25389 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:30:02.425729 25389 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:44489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:42539
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:42539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25389
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:02.426671 25389 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:02.427209 25389 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 23:30:02.433566 25389 server_base.cc:1034] running on GCE node
W20260618 23:30:02.433488 25394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:02.433732 25397 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:02.433488 25395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:02.434120 25389 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:02.434460 25389 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:02.435671 25389 hybrid_clock.cc:648] HybridClock initialized: now 1781825402435635 us; error 57 us; skew 500 ppm
I20260618 23:30:02.437404 25389 webserver.cc:460] Webserver started at http://127.8.39.190:37325/ using document root <none> and password file <none>
I20260618 23:30:02.437691 25389 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:02.437772 25389 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:02.437927 25389 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:02.439066 25389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "7a982a67646c4460aa0b294e17ada4a6"
format_stamp: "Formatted at 2026-06-18 23:30:02 on dist-test-slave-47qt"
server_key: "71be1c9365620162980d611c5f12d556"
server_key_iv: "618f788df927bae8c586cd4574f61af5"
server_key_version: "encryptionkey@0"
I20260618 23:30:02.439430 25389 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "7a982a67646c4460aa0b294e17ada4a6"
format_stamp: "Formatted at 2026-06-18 23:30:02 on dist-test-slave-47qt"
server_key: "71be1c9365620162980d611c5f12d556"
server_key_iv: "618f788df927bae8c586cd4574f61af5"
server_key_version: "encryptionkey@0"
I20260618 23:30:02.440768 25389 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:30:02.441679 25403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:02.442036 25389 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260618 23:30:02.442148 25389 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "7a982a67646c4460aa0b294e17ada4a6"
format_stamp: "Formatted at 2026-06-18 23:30:02 on dist-test-slave-47qt"
server_key: "71be1c9365620162980d611c5f12d556"
server_key_iv: "618f788df927bae8c586cd4574f61af5"
server_key_version: "encryptionkey@0"
I20260618 23:30:02.442224 25389 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:02.452194 25389 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:02.452534 25389 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:02.452690 25389 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:02.456907 25389 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:42539
I20260618 23:30:02.456943 25455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:42539 every 8 connection(s)
I20260618 23:30:02.457352 25389 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:30:02.458521 25456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:02.460884 25456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a982a67646c4460aa0b294e17ada4a6: Bootstrap starting.
I20260618 23:30:02.461550 25456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a982a67646c4460aa0b294e17ada4a6: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:02.461855 25456 log.cc:824] T 00000000000000000000000000000000 P 7a982a67646c4460aa0b294e17ada4a6: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:02.462846 25456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a982a67646c4460aa0b294e17ada4a6: No bootstrap required, opened a new log
I20260618 23:30:02.465811 25456 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a982a67646c4460aa0b294e17ada4a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a982a67646c4460aa0b294e17ada4a6" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 42539 } }
I20260618 23:30:02.465945 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 25389
I20260618 23:30:02.465997 25456 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a982a67646c4460aa0b294e17ada4a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:02.466037 25456 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a982a67646c4460aa0b294e17ada4a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a982a67646c4460aa0b294e17ada4a6, State: Initialized, Role: FOLLOWER
I20260618 23:30:02.466030 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:30:02.466243 8350 external_mini_cluster.cc:1428] Setting key 5b9436b94f482b48b2274b367538ff7c
I20260618 23:30:02.466224 25456 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a982a67646c4460aa0b294e17ada4a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a982a67646c4460aa0b294e17ada4a6" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 42539 } }
I20260618 23:30:02.466301 25456 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a982a67646c4460aa0b294e17ada4a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:30:02.466347 25456 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a982a67646c4460aa0b294e17ada4a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:30:02.466399 25456 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a982a67646c4460aa0b294e17ada4a6 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:02.467202 25456 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a982a67646c4460aa0b294e17ada4a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a982a67646c4460aa0b294e17ada4a6" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 42539 } }
I20260618 23:30:02.467314 25456 leader_election.cc:302] T 00000000000000000000000000000000 P 7a982a67646c4460aa0b294e17ada4a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a982a67646c4460aa0b294e17ada4a6; no voters:
I20260618 23:30:02.467451 25456 leader_election.cc:288] T 00000000000000000000000000000000 P 7a982a67646c4460aa0b294e17ada4a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:30:02.467535 25461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a982a67646c4460aa0b294e17ada4a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:30:02.467706 25461 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a982a67646c4460aa0b294e17ada4a6 [term 1 LEADER]: Becoming Leader. State: Replica: 7a982a67646c4460aa0b294e17ada4a6, State: Running, Role: LEADER
I20260618 23:30:02.467738 25456 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a982a67646c4460aa0b294e17ada4a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:30:02.467788 25461 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a982a67646c4460aa0b294e17ada4a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a982a67646c4460aa0b294e17ada4a6" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 42539 } }
I20260618 23:30:02.468102 25461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a982a67646c4460aa0b294e17ada4a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a982a67646c4460aa0b294e17ada4a6. Latest consensus state: current_term: 1 leader_uuid: "7a982a67646c4460aa0b294e17ada4a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a982a67646c4460aa0b294e17ada4a6" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 42539 } } }
I20260618 23:30:02.468190 25461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a982a67646c4460aa0b294e17ada4a6 [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:02.468113 25462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a982a67646c4460aa0b294e17ada4a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a982a67646c4460aa0b294e17ada4a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a982a67646c4460aa0b294e17ada4a6" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 42539 } } }
I20260618 23:30:02.468333 25462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a982a67646c4460aa0b294e17ada4a6 [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:02.468397 25467 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:30:02.468900 25467 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:30:02.470875 25467 catalog_manager.cc:1347] Generated new cluster ID: 39de351a249e4646ac78a3378e253062
I20260618 23:30:02.470938 25467 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:30:02.473289 25460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:02.466821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39798 (local address 127.8.39.190:42539)
0618 23:30:02.467063 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:02.467073 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:02.467112 (+ 39us) server_negotiation.cc:407] Connection header received
0618 23:30:02.467233 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:02.467240 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:02.467325 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:02.467476 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:02.467840 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.469721 (+ 1881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.470817 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.471435 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.471553 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:02.472436 (+ 883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:02.472440 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:02.472440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:02.472646 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:02.472686 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:02.472700 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:02.472759 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:02.472960 (+ 201us) server_negotiation.cc:299] Negotiation successful
0618 23:30:02.473041 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":87,"threads_started":1}
I20260618 23:30:02.481799 25467 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:30:02.482376 25467 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:30:02.484308 25467 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a982a67646c4460aa0b294e17ada4a6: Generated new TSK 0
I20260618 23:30:02.484691 25467 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:30:02.489890 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.129:0
--local_ip_for_outbound_sockets=127.8.39.129
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:42539
--builtin_ntp_servers=127.8.39.148:44489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 23:30:02.588281 25480 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:02.588496 25480 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:02.588526 25480 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:02.590559 25480 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:02.590663 25480 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:02.590703 25480 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.129
I20260618 23:30:02.593019 25480 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:44489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:42539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25480
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.129
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:02.593384 25480 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:02.593726 25480 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:02.600349 25488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:02.600451 25485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:02.600446 25486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:02.600850 25480 server_base.cc:1034] running on GCE node
I20260618 23:30:02.601082 25480 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:02.601375 25480 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:02.602813 25480 hybrid_clock.cc:648] HybridClock initialized: now 1781825402602784 us; error 41 us; skew 500 ppm
I20260618 23:30:02.604199 25480 webserver.cc:460] Webserver started at http://127.8.39.129:34123/ using document root <none> and password file <none>
I20260618 23:30:02.604452 25480 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:02.604534 25480 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:02.604682 25480 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:02.605827 25480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-0/data/instance:
uuid: "c675753b964a41eaadfc93c7561fbab1"
format_stamp: "Formatted at 2026-06-18 23:30:02 on dist-test-slave-47qt"
server_key: "363b6b27da8bd6ccd99166e801a1f521"
server_key_iv: "c82c08b6c7c04dcef655e7e07f87d846"
server_key_version: "encryptionkey@0"
I20260618 23:30:02.606230 25480 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance:
uuid: "c675753b964a41eaadfc93c7561fbab1"
format_stamp: "Formatted at 2026-06-18 23:30:02 on dist-test-slave-47qt"
server_key: "363b6b27da8bd6ccd99166e801a1f521"
server_key_iv: "c82c08b6c7c04dcef655e7e07f87d846"
server_key_version: "encryptionkey@0"
I20260618 23:30:02.608296 25480 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260618 23:30:02.609486 25494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:02.609724 25480 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:30:02.609838 25480 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-0/wal
uuid: "c675753b964a41eaadfc93c7561fbab1"
format_stamp: "Formatted at 2026-06-18 23:30:02 on dist-test-slave-47qt"
server_key: "363b6b27da8bd6ccd99166e801a1f521"
server_key_iv: "c82c08b6c7c04dcef655e7e07f87d846"
server_key_version: "encryptionkey@0"
I20260618 23:30:02.609936 25480 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:02.618924 25480 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:02.619256 25480 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:02.619410 25480 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:02.619648 25480 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:30:02.619987 25480 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:30:02.620041 25480 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:02.620085 25480 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:30:02.620117 25480 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:02.626014 25480 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.129:38279
I20260618 23:30:02.626149 25607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.129:38279 every 8 connection(s)
I20260618 23:30:02.626641 25480 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
I20260618 23:30:02.627094 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 25480
I20260618 23:30:02.627192 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance
I20260618 23:30:02.627488 8350 external_mini_cluster.cc:1428] Setting key 1c11410df0a1fce6f3bb4cc22b8bdf0b
I20260618 23:30:02.628783 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.130:0
--local_ip_for_outbound_sockets=127.8.39.130
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:42539
--builtin_ntp_servers=127.8.39.148:44489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260618 23:30:02.634521 25460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:02.627872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:53981 (local address 127.8.39.190:42539)
0618 23:30:02.628221 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:02.628223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:02.628240 (+ 17us) server_negotiation.cc:407] Connection header received
0618 23:30:02.628757 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:02.628759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:02.628777 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:02.628899 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:02.630110 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.630853 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.632126 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.632634 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.632800 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:02.634084 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:02.634087 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:02.634087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:02.634095 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:02.634125 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:02.634130 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:02.634223 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:02.634346 (+ 123us) server_negotiation.cc:299] Negotiation successful
0618 23:30:02.634406 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319}
I20260618 23:30:02.635001 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:02.627789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:42539 (local address 127.8.39.129:53981)
0618 23:30:02.628030 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:02.628074 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:02.628639 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:02.629050 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:02.629056 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:02.629068 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:02.629963 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:02.629974 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.630985 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.630986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:02.631996 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:02.632001 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.632823 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.632825 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:02.633018 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:02.633822 (+ 804us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:30:02.633823 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:02.633873 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:30:02.633912 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:02.634230 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:02.634238 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:02.634242 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:30:02.634352 (+ 110us) client_negotiation.cc:240] Negotiation successful
0618 23:30:02.634519 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":143,"threads_started":1}
I20260618 23:30:02.635644 25608 heartbeater.cc:347] Connected to a master server at 127.8.39.190:42539
I20260618 23:30:02.635775 25608 heartbeater.cc:464] Registering TS with master...
I20260618 23:30:02.636056 25608 heartbeater.cc:511] Master 127.8.39.190:42539 requested a full tablet report, sending...
I20260618 23:30:02.636641 25420 ts_manager.cc:194] Registered new tserver with Master: c675753b964a41eaadfc93c7561fbab1 (127.8.39.129:38279)
I20260618 23:30:02.637396 25420 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.39.129:53981
W20260618 23:30:02.720355 25611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:02.720584 25611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:02.720649 25611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:02.722009 25611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:02.722054 25611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:02.722083 25611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.130
I20260618 23:30:02.723505 25611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:44489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:42539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.130
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:02.723706 25611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:02.723974 25611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:02.729327 25619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:02.729341 25617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:02.729336 25616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:02.729723 25611 server_base.cc:1034] running on GCE node
I20260618 23:30:02.729947 25611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:02.730226 25611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:02.731459 25611 hybrid_clock.cc:648] HybridClock initialized: now 1781825402731424 us; error 35 us; skew 500 ppm
I20260618 23:30:02.733632 25611 webserver.cc:460] Webserver started at http://127.8.39.130:42985/ using document root <none> and password file <none>
I20260618 23:30:02.733986 25611 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:02.734052 25611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:02.734207 25611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:02.735678 25611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-1/data/instance:
uuid: "e4a65afa57b9464e822af847e47777e1"
format_stamp: "Formatted at 2026-06-18 23:30:02 on dist-test-slave-47qt"
server_key: "5e301f455ad93f2a0a97ca0baa90f38e"
server_key_iv: "1a58368b898fee1b1a7bc0d9f63f03bf"
server_key_version: "encryptionkey@0"
I20260618 23:30:02.736617 25611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance:
uuid: "e4a65afa57b9464e822af847e47777e1"
format_stamp: "Formatted at 2026-06-18 23:30:02 on dist-test-slave-47qt"
server_key: "5e301f455ad93f2a0a97ca0baa90f38e"
server_key_iv: "1a58368b898fee1b1a7bc0d9f63f03bf"
server_key_version: "encryptionkey@0"
I20260618 23:30:02.739135 25611 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.004s
I20260618 23:30:02.740444 25625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:02.740792 25611 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:30:02.740952 25611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-1/wal
uuid: "e4a65afa57b9464e822af847e47777e1"
format_stamp: "Formatted at 2026-06-18 23:30:02 on dist-test-slave-47qt"
server_key: "5e301f455ad93f2a0a97ca0baa90f38e"
server_key_iv: "1a58368b898fee1b1a7bc0d9f63f03bf"
server_key_version: "encryptionkey@0"
I20260618 23:30:02.741056 25611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:02.766297 25611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:02.766646 25611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:02.766805 25611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:02.767081 25611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:30:02.767496 25611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:30:02.767565 25611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:02.767609 25611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:30:02.767639 25611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:02.775799 25611 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.130:37703
I20260618 23:30:02.775846 25738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.130:37703 every 8 connection(s)
I20260618 23:30:02.776398 25611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
I20260618 23:30:02.782330 25460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:02.777383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:48625 (local address 127.8.39.190:42539)
0618 23:30:02.777458 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:02.777460 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:02.777805 (+ 345us) server_negotiation.cc:407] Connection header received
0618 23:30:02.778347 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:02.778349 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:02.778360 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:02.778520 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:02.779227 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.779751 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.780784 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.780990 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.781056 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:02.782011 (+ 955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:02.782012 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:02.782012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:02.782017 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:02.782032 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:02.782037 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:02.782089 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:02.782236 (+ 147us) server_negotiation.cc:299] Negotiation successful
0618 23:30:02.782262 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260618 23:30:02.782640 25741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:02.777370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:42539 (local address 127.8.39.130:48625)
0618 23:30:02.777667 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:02.777720 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:02.778221 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:02.778591 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:02.778595 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:02.778603 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:02.779122 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:02.779126 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.779841 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.779842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:02.780691 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:02.780694 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.781080 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.781081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:02.781208 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:02.781824 (+ 616us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:30:02.781824 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:02.781854 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:30:02.781877 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:02.782124 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:02.782132 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:02.782134 (+ 2us) client_negotiation.cc:769] Sending connection context
0618 23:30:02.782218 (+ 84us) client_negotiation.cc:240] Negotiation successful
0618 23:30:02.782330 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":123,"threads_started":1}
I20260618 23:30:02.783308 25739 heartbeater.cc:347] Connected to a master server at 127.8.39.190:42539
I20260618 23:30:02.783401 25739 heartbeater.cc:464] Registering TS with master...
I20260618 23:30:02.783596 25739 heartbeater.cc:511] Master 127.8.39.190:42539 requested a full tablet report, sending...
I20260618 23:30:02.784185 25420 ts_manager.cc:194] Registered new tserver with Master: e4a65afa57b9464e822af847e47777e1 (127.8.39.130:37703)
I20260618 23:30:02.784771 25420 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.39.130:48625
I20260618 23:30:02.786491 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 25611
I20260618 23:30:02.786618 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance
I20260618 23:30:02.786849 8350 external_mini_cluster.cc:1428] Setting key 741a356f70f3150020bde02180bad9a4
I20260618 23:30:02.788118 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.131:0
--local_ip_for_outbound_sockets=127.8.39.131
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:42539
--builtin_ntp_servers=127.8.39.148:44489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 23:30:02.874940 25742 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:02.875144 25742 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:02.875166 25742 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:02.876380 25742 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:02.876425 25742 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:02.876456 25742 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.131
I20260618 23:30:02.877889 25742 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:44489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:42539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25742
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.131
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:02.878134 25742 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:02.878435 25742 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:02.883692 25747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:02.883757 25748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:02.883755 25750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:02.884032 25742 server_base.cc:1034] running on GCE node
I20260618 23:30:02.884256 25742 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:02.884516 25742 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:02.885697 25742 hybrid_clock.cc:648] HybridClock initialized: now 1781825402885685 us; error 31 us; skew 500 ppm
I20260618 23:30:02.887142 25742 webserver.cc:460] Webserver started at http://127.8.39.131:40589/ using document root <none> and password file <none>
I20260618 23:30:02.887398 25742 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:02.887472 25742 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:02.887620 25742 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:02.888711 25742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-2/data/instance:
uuid: "3facef28dc94475aac9cdc773a296c5a"
format_stamp: "Formatted at 2026-06-18 23:30:02 on dist-test-slave-47qt"
server_key: "89b9e646db83b76a04ffbd9f75292429"
server_key_iv: "f74440a44f6f15f46bf45043276804a6"
server_key_version: "encryptionkey@0"
I20260618 23:30:02.889082 25742 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance:
uuid: "3facef28dc94475aac9cdc773a296c5a"
format_stamp: "Formatted at 2026-06-18 23:30:02 on dist-test-slave-47qt"
server_key: "89b9e646db83b76a04ffbd9f75292429"
server_key_iv: "f74440a44f6f15f46bf45043276804a6"
server_key_version: "encryptionkey@0"
I20260618 23:30:02.890491 25742 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:30:02.891610 25756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:02.891908 25742 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:30:02.892032 25742 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-2/wal
uuid: "3facef28dc94475aac9cdc773a296c5a"
format_stamp: "Formatted at 2026-06-18 23:30:02 on dist-test-slave-47qt"
server_key: "89b9e646db83b76a04ffbd9f75292429"
server_key_iv: "f74440a44f6f15f46bf45043276804a6"
server_key_version: "encryptionkey@0"
I20260618 23:30:02.892118 25742 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:02.915165 25742 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:02.915508 25742 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:02.915669 25742 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:02.915925 25742 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:30:02.916435 25742 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:30:02.916586 25742 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:02.916632 25742 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:30:02.916661 25742 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:02.924362 25742 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.131:39215
I20260618 23:30:02.924405 25869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.131:39215 every 8 connection(s)
I20260618 23:30:02.924778 25742 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
I20260618 23:30:02.930646 25460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:02.925647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:48143 (local address 127.8.39.190:42539)
0618 23:30:02.925722 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:02.925725 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:02.925915 (+ 190us) server_negotiation.cc:407] Connection header received
0618 23:30:02.926406 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:02.926407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:02.926418 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:02.926506 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:02.927222 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.927854 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.928864 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.929076 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.929210 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:02.930145 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:02.930147 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:02.930147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:02.930157 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:02.930194 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:02.930200 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:02.930295 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:02.930547 (+ 252us) server_negotiation.cc:299] Negotiation successful
0618 23:30:02.930578 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260618 23:30:02.931016 25872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:02.925555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:42539 (local address 127.8.39.131:48143)
0618 23:30:02.925772 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:02.925814 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:02.926286 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:02.926606 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:02.926610 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:02.926618 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:02.927112 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:02.927116 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.927967 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.927968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:02.928778 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:02.928780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.929217 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.929218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:02.929342 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:02.929900 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:30:02.929900 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:02.929932 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:30:02.929958 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:02.930362 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:02.930378 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:02.930384 (+ 6us) client_negotiation.cc:769] Sending connection context
0618 23:30:02.930497 (+ 113us) client_negotiation.cc:240] Negotiation successful
0618 23:30:02.930657 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":84,"threads_started":1}
I20260618 23:30:02.932117 25870 heartbeater.cc:347] Connected to a master server at 127.8.39.190:42539
I20260618 23:30:02.932363 25870 heartbeater.cc:464] Registering TS with master...
I20260618 23:30:02.932768 25870 heartbeater.cc:511] Master 127.8.39.190:42539 requested a full tablet report, sending...
I20260618 23:30:02.933652 25420 ts_manager.cc:194] Registered new tserver with Master: 3facef28dc94475aac9cdc773a296c5a (127.8.39.131:39215)
I20260618 23:30:02.934098 25420 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.39.131:48143
I20260618 23:30:02.934948 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 25742
I20260618 23:30:02.935068 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance
I20260618 23:30:02.935288 8350 external_mini_cluster.cc:1428] Setting key a393cc6cf1a99d402ed597b55f030e03
I20260618 23:30:02.936553 8350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:30:02.942178 25460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:02.938742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39806 (local address 127.8.39.190:42539)
0618 23:30:02.938818 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:02.938820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:02.938847 (+ 27us) server_negotiation.cc:407] Connection header received
0618 23:30:02.938954 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:02.938955 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:02.938972 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:02.939079 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:02.939264 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.939889 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.940712 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.940900 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.940987 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:02.941890 (+ 903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:02.941892 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:02.941892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:02.941897 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:02.941917 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:02.941921 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:02.941985 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:02.942052 (+ 67us) server_negotiation.cc:299] Negotiation successful
0618 23:30:02.942077 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260618 23:30:02.944671 25420 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39806:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:30:02.947358 25420 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:30:02.956136 25878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:02.950691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:37703 (local address 127.0.0.1:40276)
0618 23:30:02.950865 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:02.950905 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:02.950992 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:02.951773 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:02.951774 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:02.951776 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:30:02.951815 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:02.951816 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.953380 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.953382 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:02.954375 (+ 993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:02.954378 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.955663 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.955664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:02.955798 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:02.955826 (+ 28us) client_negotiation.cc:769] Sending connection context
0618 23:30:02.955929 (+ 103us) client_negotiation.cc:240] Negotiation successful
0618 23:30:02.956024 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":6,"spinlock_wait_cycles":1280,"thread_start_us":87,"threads_started":1}
I20260618 23:30:02.956159 25882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:02.951006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46396 (local address 127.8.39.129:38279)
0618 23:30:02.951445 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:02.951449 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:02.951483 (+ 34us) server_negotiation.cc:407] Connection header received
0618 23:30:02.951533 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:02.951535 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:02.951552 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:02.951823 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:30:02.951883 (+ 60us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.952638 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.954466 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.955512 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.955604 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:02.955701 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:02.956007 (+ 306us) server_negotiation.cc:299] Negotiation successful
0618 23:30:02.956072 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":401,"thread_start_us":124,"threads_started":1}
I20260618 23:30:02.956964 25879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:02.950670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:38279 (local address 127.0.0.1:46396)
0618 23:30:02.950864 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:02.950933 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:02.950992 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:02.951677 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:02.951678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:02.951680 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:30:02.951739 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:02.951741 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.952739 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.952740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:02.954345 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:02.954346 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.955665 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.955665 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:02.955798 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:02.955826 (+ 28us) client_negotiation.cc:769] Sending connection context
0618 23:30:02.955929 (+ 103us) client_negotiation.cc:240] Negotiation successful
0618 23:30:02.956865 (+ 936us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":883,"thread_start_us":63,"threads_started":1}
I20260618 23:30:02.957132 25880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:02.950691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:39215 (local address 127.0.0.1:40458)
0618 23:30:02.951102 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:02.951111 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:02.951149 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:02.951419 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:02.951423 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:02.951425 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:30:02.951504 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:02.951506 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.953031 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.953031 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:02.954338 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:02.954341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.956739 (+ 2398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.956741 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:02.956819 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:02.956826 (+ 7us) client_negotiation.cc:769] Sending connection context
0618 23:30:02.956900 (+ 74us) client_negotiation.cc:240] Negotiation successful
0618 23:30:02.956939 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":394,"mutex_wait_us":5,"thread_start_us":58,"threads_started":1}
I20260618 23:30:02.957120 25881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:02.950987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40458 (local address 127.8.39.131:39215)
0618 23:30:02.951214 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:02.951219 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:02.951230 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:30:02.951279 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:02.951281 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:02.951301 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:02.951396 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:30:02.951595 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.952900 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.955754 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.956614 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.956687 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:02.956748 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:02.956979 (+ 231us) server_negotiation.cc:299] Negotiation successful
0618 23:30:02.957015 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":114,"threads_started":1}
I20260618 23:30:02.957640 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:02.951230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40276 (local address 127.8.39.130:37703)
0618 23:30:02.951605 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:02.951610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:02.951619 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:30:02.951658 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:02.951660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:02.951674 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:02.951883 (+ 209us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:30:02.951932 (+ 49us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.953278 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.954518 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:02.955475 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:02.955583 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:02.955702 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:02.957508 (+ 1806us) server_negotiation.cc:299] Negotiation successful
0618 23:30:02.957559 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":97,"threads_started":1}
I20260618 23:30:02.957927 25804 tablet_service.cc:1467] Processing CreateTablet for tablet 1f131ad1a17d4de2ab076608149ddf76 (DEFAULT_TABLE table=test-table [id=c322270853ec4904bfefb40d4219e4e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:30:02.957933 25542 tablet_service.cc:1467] Processing CreateTablet for tablet 1f131ad1a17d4de2ab076608149ddf76 (DEFAULT_TABLE table=test-table [id=c322270853ec4904bfefb40d4219e4e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:30:02.958312 25804 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f131ad1a17d4de2ab076608149ddf76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:02.958318 25542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f131ad1a17d4de2ab076608149ddf76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:02.958220 25673 tablet_service.cc:1467] Processing CreateTablet for tablet 1f131ad1a17d4de2ab076608149ddf76 (DEFAULT_TABLE table=test-table [id=c322270853ec4904bfefb40d4219e4e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:30:02.958530 25673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f131ad1a17d4de2ab076608149ddf76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:02.961148 25884 tablet_bootstrap.cc:492] T 1f131ad1a17d4de2ab076608149ddf76 P 3facef28dc94475aac9cdc773a296c5a: Bootstrap starting.
I20260618 23:30:02.961757 25884 tablet_bootstrap.cc:654] T 1f131ad1a17d4de2ab076608149ddf76 P 3facef28dc94475aac9cdc773a296c5a: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:02.961907 25885 tablet_bootstrap.cc:492] T 1f131ad1a17d4de2ab076608149ddf76 P e4a65afa57b9464e822af847e47777e1: Bootstrap starting.
I20260618 23:30:02.962155 25886 tablet_bootstrap.cc:492] T 1f131ad1a17d4de2ab076608149ddf76 P c675753b964a41eaadfc93c7561fbab1: Bootstrap starting.
I20260618 23:30:02.962308 25884 log.cc:824] T 1f131ad1a17d4de2ab076608149ddf76 P 3facef28dc94475aac9cdc773a296c5a: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:02.962581 25885 tablet_bootstrap.cc:654] T 1f131ad1a17d4de2ab076608149ddf76 P e4a65afa57b9464e822af847e47777e1: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:02.962880 25885 log.cc:824] T 1f131ad1a17d4de2ab076608149ddf76 P e4a65afa57b9464e822af847e47777e1: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:02.962904 25886 tablet_bootstrap.cc:654] T 1f131ad1a17d4de2ab076608149ddf76 P c675753b964a41eaadfc93c7561fbab1: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:02.963205 25886 log.cc:824] T 1f131ad1a17d4de2ab076608149ddf76 P c675753b964a41eaadfc93c7561fbab1: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:02.963716 25885 tablet_bootstrap.cc:492] T 1f131ad1a17d4de2ab076608149ddf76 P e4a65afa57b9464e822af847e47777e1: No bootstrap required, opened a new log
I20260618 23:30:02.963783 25885 ts_tablet_manager.cc:1397] T 1f131ad1a17d4de2ab076608149ddf76 P e4a65afa57b9464e822af847e47777e1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:30:02.964203 25884 tablet_bootstrap.cc:492] T 1f131ad1a17d4de2ab076608149ddf76 P 3facef28dc94475aac9cdc773a296c5a: No bootstrap required, opened a new log
I20260618 23:30:02.964319 25884 ts_tablet_manager.cc:1397] T 1f131ad1a17d4de2ab076608149ddf76 P 3facef28dc94475aac9cdc773a296c5a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260618 23:30:02.964443 25886 tablet_bootstrap.cc:492] T 1f131ad1a17d4de2ab076608149ddf76 P c675753b964a41eaadfc93c7561fbab1: No bootstrap required, opened a new log
I20260618 23:30:02.964514 25886 ts_tablet_manager.cc:1397] T 1f131ad1a17d4de2ab076608149ddf76 P c675753b964a41eaadfc93c7561fbab1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:30:02.966424 25885 raft_consensus.cc:348] T 1f131ad1a17d4de2ab076608149ddf76 P e4a65afa57b9464e822af847e47777e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3facef28dc94475aac9cdc773a296c5a" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 39215 } } peers { permanent_uuid: "c675753b964a41eaadfc93c7561fbab1" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38279 } } peers { permanent_uuid: "e4a65afa57b9464e822af847e47777e1" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37703 } }
I20260618 23:30:02.966864 25885 raft_consensus.cc:374] T 1f131ad1a17d4de2ab076608149ddf76 P e4a65afa57b9464e822af847e47777e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:02.966926 25885 raft_consensus.cc:729] T 1f131ad1a17d4de2ab076608149ddf76 P e4a65afa57b9464e822af847e47777e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4a65afa57b9464e822af847e47777e1, State: Initialized, Role: FOLLOWER
I20260618 23:30:02.967082 25885 consensus_queue.cc:260] T 1f131ad1a17d4de2ab076608149ddf76 P e4a65afa57b9464e822af847e47777e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3facef28dc94475aac9cdc773a296c5a" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 39215 } } peers { permanent_uuid: "c675753b964a41eaadfc93c7561fbab1" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38279 } } peers { permanent_uuid: "e4a65afa57b9464e822af847e47777e1" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37703 } }
I20260618 23:30:02.967236 25884 raft_consensus.cc:348] T 1f131ad1a17d4de2ab076608149ddf76 P 3facef28dc94475aac9cdc773a296c5a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3facef28dc94475aac9cdc773a296c5a" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 39215 } } peers { permanent_uuid: "c675753b964a41eaadfc93c7561fbab1" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38279 } } peers { permanent_uuid: "e4a65afa57b9464e822af847e47777e1" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37703 } }
I20260618 23:30:02.967514 25884 raft_consensus.cc:374] T 1f131ad1a17d4de2ab076608149ddf76 P 3facef28dc94475aac9cdc773a296c5a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:02.967566 25884 raft_consensus.cc:729] T 1f131ad1a17d4de2ab076608149ddf76 P 3facef28dc94475aac9cdc773a296c5a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3facef28dc94475aac9cdc773a296c5a, State: Initialized, Role: FOLLOWER
I20260618 23:30:02.967182 25886 raft_consensus.cc:348] T 1f131ad1a17d4de2ab076608149ddf76 P c675753b964a41eaadfc93c7561fbab1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3facef28dc94475aac9cdc773a296c5a" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 39215 } } peers { permanent_uuid: "c675753b964a41eaadfc93c7561fbab1" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38279 } } peers { permanent_uuid: "e4a65afa57b9464e822af847e47777e1" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37703 } }
I20260618 23:30:02.967700 25885 ts_tablet_manager.cc:1428] T 1f131ad1a17d4de2ab076608149ddf76 P e4a65afa57b9464e822af847e47777e1: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260618 23:30:02.967729 25886 raft_consensus.cc:374] T 1f131ad1a17d4de2ab076608149ddf76 P c675753b964a41eaadfc93c7561fbab1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:02.967825 25886 raft_consensus.cc:729] T 1f131ad1a17d4de2ab076608149ddf76 P c675753b964a41eaadfc93c7561fbab1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c675753b964a41eaadfc93c7561fbab1, State: Initialized, Role: FOLLOWER
I20260618 23:30:02.967916 25739 heartbeater.cc:503] Master 127.8.39.190:42539 was elected leader, sending a full tablet report...
I20260618 23:30:02.968044 25886 consensus_queue.cc:260] T 1f131ad1a17d4de2ab076608149ddf76 P c675753b964a41eaadfc93c7561fbab1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3facef28dc94475aac9cdc773a296c5a" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 39215 } } peers { permanent_uuid: "c675753b964a41eaadfc93c7561fbab1" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38279 } } peers { permanent_uuid: "e4a65afa57b9464e822af847e47777e1" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37703 } }
I20260618 23:30:02.968488 25884 consensus_queue.cc:260] T 1f131ad1a17d4de2ab076608149ddf76 P 3facef28dc94475aac9cdc773a296c5a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3facef28dc94475aac9cdc773a296c5a" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 39215 } } peers { permanent_uuid: "c675753b964a41eaadfc93c7561fbab1" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38279 } } peers { permanent_uuid: "e4a65afa57b9464e822af847e47777e1" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37703 } }
I20260618 23:30:02.968770 25608 heartbeater.cc:503] Master 127.8.39.190:42539 was elected leader, sending a full tablet report...
I20260618 23:30:02.968788 25886 ts_tablet_manager.cc:1428] T 1f131ad1a17d4de2ab076608149ddf76 P c675753b964a41eaadfc93c7561fbab1: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260618 23:30:02.968844 25884 ts_tablet_manager.cc:1428] T 1f131ad1a17d4de2ab076608149ddf76 P 3facef28dc94475aac9cdc773a296c5a: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260618 23:30:02.968956 25870 heartbeater.cc:503] Master 127.8.39.190:42539 was elected leader, sending a full tablet report...
I20260618 23:30:03.026712 25891 raft_consensus.cc:482] T 1f131ad1a17d4de2ab076608149ddf76 P c675753b964a41eaadfc93c7561fbab1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:30:03.027076 25891 raft_consensus.cc:504] T 1f131ad1a17d4de2ab076608149ddf76 P c675753b964a41eaadfc93c7561fbab1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3facef28dc94475aac9cdc773a296c5a" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 39215 } } peers { permanent_uuid: "c675753b964a41eaadfc93c7561fbab1" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38279 } } peers { permanent_uuid: "e4a65afa57b9464e822af847e47777e1" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37703 } }
W20260618 23:30:03.027191 25740 tablet.cc:2369] T 1f131ad1a17d4de2ab076608149ddf76 P e4a65afa57b9464e822af847e47777e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:30:03.027599 25891 leader_election.cc:288] T 1f131ad1a17d4de2ab076608149ddf76 P c675753b964a41eaadfc93c7561fbab1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3facef28dc94475aac9cdc773a296c5a (127.8.39.131:39215), e4a65afa57b9464e822af847e47777e1 (127.8.39.130:37703)
I20260618 23:30:03.030552 25883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:03.027873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:50129 (local address 127.8.39.130:37703)
0618 23:30:03.027914 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:03.027916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:03.027977 (+ 61us) server_negotiation.cc:407] Connection header received
0618 23:30:03.028056 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:03.028057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:03.028068 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:03.028142 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:30:03.028293 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:03.028762 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:03.029804 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:03.030186 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:03.030271 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:03.030279 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:03.030445 (+ 166us) server_negotiation.cc:299] Negotiation successful
0618 23:30:03.030482 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260618 23:30:03.030568 25893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:03.027778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:37703 (local address 127.8.39.129:50129)
0618 23:30:03.027920 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:03.027931 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:03.027993 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:03.028146 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:03.028147 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:03.028149 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:30:03.028217 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:03.028218 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:03.028847 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:03.028848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:03.029707 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:03.029709 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:03.030280 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:03.030281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:03.030366 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:03.030370 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:30:03.030438 (+ 68us) client_negotiation.cc:240] Negotiation successful
0618 23:30:03.030479 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
I20260618 23:30:03.030642 25881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:03.027796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:54213 (local address 127.8.39.131:39215)
0618 23:30:03.027850 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:03.027852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:03.027916 (+ 64us) server_negotiation.cc:407] Connection header received
0618 23:30:03.027962 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:03.027963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:03.027983 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:03.028054 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:30:03.028224 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:03.028735 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:03.029828 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:03.030274 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:03.030374 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:03.030383 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:03.030530 (+ 147us) server_negotiation.cc:299] Negotiation successful
0618 23:30:03.030570 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260618 23:30:03.030656 25610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:03.027744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:39215 (local address 127.8.39.129:54213)
0618 23:30:03.027823 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:03.027838 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:03.027901 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:03.028073 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:03.028075 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:03.028076 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:30:03.028148 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:03.028150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:03.028837 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:03.028838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:03.029689 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:03.029692 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:03.030374 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:03.030375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:03.030446 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:03.030449 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:30:03.030512 (+ 63us) client_negotiation.cc:240] Negotiation successful
0618 23:30:03.030565 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":1}
I20260618 23:30:03.030923 25693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f131ad1a17d4de2ab076608149ddf76" candidate_uuid: "c675753b964a41eaadfc93c7561fbab1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4a65afa57b9464e822af847e47777e1" is_pre_election: true
I20260618 23:30:03.031105 25693 raft_consensus.cc:2454] T 1f131ad1a17d4de2ab076608149ddf76 P e4a65afa57b9464e822af847e47777e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c675753b964a41eaadfc93c7561fbab1 in term 0.
I20260618 23:30:03.031112 25824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f131ad1a17d4de2ab076608149ddf76" candidate_uuid: "c675753b964a41eaadfc93c7561fbab1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3facef28dc94475aac9cdc773a296c5a" is_pre_election: true
I20260618 23:30:03.031270 25824 raft_consensus.cc:2454] T 1f131ad1a17d4de2ab076608149ddf76 P 3facef28dc94475aac9cdc773a296c5a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c675753b964a41eaadfc93c7561fbab1 in term 0.
I20260618 23:30:03.031380 25498 leader_election.cc:302] T 1f131ad1a17d4de2ab076608149ddf76 P c675753b964a41eaadfc93c7561fbab1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c675753b964a41eaadfc93c7561fbab1, e4a65afa57b9464e822af847e47777e1; no voters:
I20260618 23:30:03.031510 25891 raft_consensus.cc:2781] T 1f131ad1a17d4de2ab076608149ddf76 P c675753b964a41eaadfc93c7561fbab1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:30:03.031569 25891 raft_consensus.cc:482] T 1f131ad1a17d4de2ab076608149ddf76 P c675753b964a41eaadfc93c7561fbab1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:30:03.031597 25891 raft_consensus.cc:3037] T 1f131ad1a17d4de2ab076608149ddf76 P c675753b964a41eaadfc93c7561fbab1 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:03.032271 25891 raft_consensus.cc:504] T 1f131ad1a17d4de2ab076608149ddf76 P c675753b964a41eaadfc93c7561fbab1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3facef28dc94475aac9cdc773a296c5a" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 39215 } } peers { permanent_uuid: "c675753b964a41eaadfc93c7561fbab1" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38279 } } peers { permanent_uuid: "e4a65afa57b9464e822af847e47777e1" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37703 } }
I20260618 23:30:03.032420 25891 leader_election.cc:288] T 1f131ad1a17d4de2ab076608149ddf76 P c675753b964a41eaadfc93c7561fbab1 [CANDIDATE]: Term 1 election: Requested vote from peers 3facef28dc94475aac9cdc773a296c5a (127.8.39.131:39215), e4a65afa57b9464e822af847e47777e1 (127.8.39.130:37703)
I20260618 23:30:03.032557 25824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f131ad1a17d4de2ab076608149ddf76" candidate_uuid: "c675753b964a41eaadfc93c7561fbab1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3facef28dc94475aac9cdc773a296c5a"
I20260618 23:30:03.032624 25693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f131ad1a17d4de2ab076608149ddf76" candidate_uuid: "c675753b964a41eaadfc93c7561fbab1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e4a65afa57b9464e822af847e47777e1"
I20260618 23:30:03.032667 25824 raft_consensus.cc:3037] T 1f131ad1a17d4de2ab076608149ddf76 P 3facef28dc94475aac9cdc773a296c5a [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:03.032691 25693 raft_consensus.cc:3037] T 1f131ad1a17d4de2ab076608149ddf76 P e4a65afa57b9464e822af847e47777e1 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:03.033591 25693 raft_consensus.cc:2454] T 1f131ad1a17d4de2ab076608149ddf76 P e4a65afa57b9464e822af847e47777e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c675753b964a41eaadfc93c7561fbab1 in term 1.
I20260618 23:30:03.033602 25824 raft_consensus.cc:2454] T 1f131ad1a17d4de2ab076608149ddf76 P 3facef28dc94475aac9cdc773a296c5a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c675753b964a41eaadfc93c7561fbab1 in term 1.
I20260618 23:30:03.033819 25495 leader_election.cc:302] T 1f131ad1a17d4de2ab076608149ddf76 P c675753b964a41eaadfc93c7561fbab1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3facef28dc94475aac9cdc773a296c5a, c675753b964a41eaadfc93c7561fbab1; no voters:
I20260618 23:30:03.033926 25891 raft_consensus.cc:2781] T 1f131ad1a17d4de2ab076608149ddf76 P c675753b964a41eaadfc93c7561fbab1 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:30:03.034046 25891 raft_consensus.cc:686] T 1f131ad1a17d4de2ab076608149ddf76 P c675753b964a41eaadfc93c7561fbab1 [term 1 LEADER]: Becoming Leader. State: Replica: c675753b964a41eaadfc93c7561fbab1, State: Running, Role: LEADER
I20260618 23:30:03.034154 25891 consensus_queue.cc:237] T 1f131ad1a17d4de2ab076608149ddf76 P c675753b964a41eaadfc93c7561fbab1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3facef28dc94475aac9cdc773a296c5a" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 39215 } } peers { permanent_uuid: "c675753b964a41eaadfc93c7561fbab1" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38279 } } peers { permanent_uuid: "e4a65afa57b9464e822af847e47777e1" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37703 } }
I20260618 23:30:03.035156 25419 catalog_manager.cc:5495] T 1f131ad1a17d4de2ab076608149ddf76 P c675753b964a41eaadfc93c7561fbab1 reported cstate change: term changed from 0 to 1, leader changed from <none> to c675753b964a41eaadfc93c7561fbab1 (127.8.39.129). New cstate: current_term: 1 leader_uuid: "c675753b964a41eaadfc93c7561fbab1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3facef28dc94475aac9cdc773a296c5a" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 39215 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c675753b964a41eaadfc93c7561fbab1" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 38279 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e4a65afa57b9464e822af847e47777e1" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37703 } health_report { overall_health: UNKNOWN } } }
I20260618 23:30:03.050774 25882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:03.048142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46408 (local address 127.8.39.129:38279)
0618 23:30:03.048253 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:03.048255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:03.048264 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:30:03.048301 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:03.048302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:03.048327 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:03.048399 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:30:03.048596 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:03.049086 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:03.049936 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:03.050135 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:03.050199 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:03.050376 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:30:03.050500 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:30:03.050590 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:03.050668 (+ 78us) server_negotiation.cc:299] Negotiation successful
0618 23:30:03.050688 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260618 23:30:03.053014 25693 raft_consensus.cc:1261] T 1f131ad1a17d4de2ab076608149ddf76 P e4a65afa57b9464e822af847e47777e1 [term 1 FOLLOWER]: Refusing update from remote peer c675753b964a41eaadfc93c7561fbab1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:30:03.053023 25824 raft_consensus.cc:1261] T 1f131ad1a17d4de2ab076608149ddf76 P 3facef28dc94475aac9cdc773a296c5a [term 1 FOLLOWER]: Refusing update from remote peer c675753b964a41eaadfc93c7561fbab1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:30:03.053436 25894 consensus_queue.cc:1035] T 1f131ad1a17d4de2ab076608149ddf76 P c675753b964a41eaadfc93c7561fbab1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3facef28dc94475aac9cdc773a296c5a" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 39215 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:30:03.053580 25891 consensus_queue.cc:1035] T 1f131ad1a17d4de2ab076608149ddf76 P c675753b964a41eaadfc93c7561fbab1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e4a65afa57b9464e822af847e47777e1" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 37703 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:30:03.056910 25899 mvcc.cc:205] Tried to move back new op lower bound from 7298356850902028288 to 7298356850828247040. Current Snapshot: MvccSnapshot[applied={T|T < 7298356850902028288}]
I20260618 23:30:03.058274 25898 mvcc.cc:205] Tried to move back new op lower bound from 7298356850902028288 to 7298356850828247040. Current Snapshot: MvccSnapshot[applied={T|T < 7298356850902028288 or (T in {7298356850902028288})}]
I20260618 23:30:03.058670 25419 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:30:03.058770 25419 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39806:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:30:03.059495 25419 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a982a67646c4460aa0b294e17ada4a6: Sending DeleteTablet for 3 replicas of tablet 1f131ad1a17d4de2ab076608149ddf76
I20260618 23:30:03.059854 25673 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f131ad1a17d4de2ab076608149ddf76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:30:03 UTC) from {username='slave'} at 127.0.0.1:40276
I20260618 23:30:03.059870 25804 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f131ad1a17d4de2ab076608149ddf76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:30:03 UTC) from {username='slave'} at 127.0.0.1:40458
I20260618 23:30:03.060156 25907 tablet_replica.cc:330] T 1f131ad1a17d4de2ab076608149ddf76 P 3facef28dc94475aac9cdc773a296c5a: stopping tablet replica
I20260618 23:30:03.060130 25542 tablet_service.cc:1514] Processing DeleteTablet for tablet 1f131ad1a17d4de2ab076608149ddf76 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:30:03 UTC) from {username='slave'} at 127.0.0.1:46396
I20260618 23:30:03.060266 25907 raft_consensus.cc:2229] T 1f131ad1a17d4de2ab076608149ddf76 P 3facef28dc94475aac9cdc773a296c5a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:30:03.060367 25907 raft_consensus.cc:2258] T 1f131ad1a17d4de2ab076608149ddf76 P 3facef28dc94475aac9cdc773a296c5a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:30:03.060364 25908 tablet_replica.cc:330] T 1f131ad1a17d4de2ab076608149ddf76 P c675753b964a41eaadfc93c7561fbab1: stopping tablet replica
I20260618 23:30:03.060465 25908 raft_consensus.cc:2229] T 1f131ad1a17d4de2ab076608149ddf76 P c675753b964a41eaadfc93c7561fbab1 [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:30:03.060527 25906 tablet_replica.cc:330] T 1f131ad1a17d4de2ab076608149ddf76 P e4a65afa57b9464e822af847e47777e1: stopping tablet replica
I20260618 23:30:03.060560 25908 raft_consensus.cc:2258] T 1f131ad1a17d4de2ab076608149ddf76 P c675753b964a41eaadfc93c7561fbab1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:30:03.060621 25906 raft_consensus.cc:2229] T 1f131ad1a17d4de2ab076608149ddf76 P e4a65afa57b9464e822af847e47777e1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:30:03.060669 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 25480
I20260618 23:30:03.060720 25906 raft_consensus.cc:2258] T 1f131ad1a17d4de2ab076608149ddf76 P e4a65afa57b9464e822af847e47777e1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:30:03.061193 25907 ts_tablet_manager.cc:1905] T 1f131ad1a17d4de2ab076608149ddf76 P 3facef28dc94475aac9cdc773a296c5a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:30:03.061789 25906 ts_tablet_manager.cc:1905] T 1f131ad1a17d4de2ab076608149ddf76 P e4a65afa57b9464e822af847e47777e1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:30:03.063277 25907 ts_tablet_manager.cc:1918] T 1f131ad1a17d4de2ab076608149ddf76 P 3facef28dc94475aac9cdc773a296c5a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:30:03.063395 25906 ts_tablet_manager.cc:1918] T 1f131ad1a17d4de2ab076608149ddf76 P e4a65afa57b9464e822af847e47777e1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:30:03.063411 25907 log.cc:1196] T 1f131ad1a17d4de2ab076608149ddf76 P 3facef28dc94475aac9cdc773a296c5a: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-2/wal/wals/1f131ad1a17d4de2ab076608149ddf76
I20260618 23:30:03.063462 25906 log.cc:1196] T 1f131ad1a17d4de2ab076608149ddf76 P e4a65afa57b9464e822af847e47777e1: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781825053542692-8350-0/minicluster-data/ts-1/wal/wals/1f131ad1a17d4de2ab076608149ddf76
I20260618 23:30:03.063722 25906 ts_tablet_manager.cc:1939] T 1f131ad1a17d4de2ab076608149ddf76 P e4a65afa57b9464e822af847e47777e1: Deleting consensus metadata
I20260618 23:30:03.063827 25907 ts_tablet_manager.cc:1939] T 1f131ad1a17d4de2ab076608149ddf76 P 3facef28dc94475aac9cdc773a296c5a: Deleting consensus metadata
I20260618 23:30:03.064198 25407 catalog_manager.cc:4841] TS e4a65afa57b9464e822af847e47777e1 (127.8.39.130:37703): tablet 1f131ad1a17d4de2ab076608149ddf76 (table test-table [id=c322270853ec4904bfefb40d4219e4e8]) successfully deleted
I20260618 23:30:03.064266 25404 catalog_manager.cc:4841] TS 3facef28dc94475aac9cdc773a296c5a (127.8.39.131:39215): tablet 1f131ad1a17d4de2ab076608149ddf76 (table test-table [id=c322270853ec4904bfefb40d4219e4e8]) successfully deleted
W20260618 23:30:03.067428 25405 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.39.129:38279 (error 108)
I20260618 23:30:03.067567 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 25611
I20260618 23:30:03.069017 25879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:03.068787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:38279 (local address 127.0.0.1:46420)
0618 23:30:03.068874 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:03.068939 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.129:38279: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260618 23:30:03.069340 25405 catalog_manager.cc:4584] TS c675753b964a41eaadfc93c7561fbab1 (127.8.39.129:38279): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1f131ad1a17d4de2ab076608149ddf76: Network error: Client connection negotiation failed: client connection to 127.8.39.129:38279: connect: Connection refused (error 111)
I20260618 23:30:03.074007 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 25742
I20260618 23:30:03.079267 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 25389
2026-06-18T23:30:03Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (779 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:30:03 dist-test-slave-47qt krb5kdc[25912](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:30:03 dist-test-slave-47qt krb5kdc[25912](info): set up 2 sockets
Jun 18 23:30:03 dist-test-slave-47qt krb5kdc[25912](info): commencing operation
krb5kdc: starting...
W20260618 23:30:04.168848 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.068s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:30:04 dist-test-slave-47qt krb5kdc[25912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:30:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:30:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.39.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
I20260618 23:30:04.299654 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:37283
--webserver_interface=127.8.39.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:45363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:37283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:30:04.389273 25925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:04.389501 25925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:04.389526 25925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:04.390825 25925 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:30:04.390863 25925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:04.390878 25925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:04.390890 25925 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:30:04.390903 25925 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:30:04.392324 25925 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:45363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:37283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:37283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:04.392525 25925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:04.392838 25925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:04.397908 25933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:04.397891 25930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:04.397869 25931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:04.397958 25925 server_base.cc:1034] running on GCE node
I20260618 23:30:04.398445 25925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:04.398804 25925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:04.399986 25925 hybrid_clock.cc:648] HybridClock initialized: now 1781825404399961 us; error 57 us; skew 500 ppm
Jun 18 23:30:04 dist-test-slave-47qt krb5kdc[25912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825404, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:04.402549 25925 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:30:04.402995 25925 webserver.cc:460] Webserver started at http://127.8.39.190:45151/ using document root <none> and password file <none>
I20260618 23:30:04.403221 25925 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:04.403290 25925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:04.403443 25925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:04.404532 25925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "edf58ae2793e418bb213d5748f8a238e"
format_stamp: "Formatted at 2026-06-18 23:30:04 on dist-test-slave-47qt"
server_key: "e0951b468aac14873f1b015de5fb73e9"
server_key_iv: "81dc44fb39c54966a85ebc8621399d0b"
server_key_version: "encryptionkey@0"
I20260618 23:30:04.404912 25925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "edf58ae2793e418bb213d5748f8a238e"
format_stamp: "Formatted at 2026-06-18 23:30:04 on dist-test-slave-47qt"
server_key: "e0951b468aac14873f1b015de5fb73e9"
server_key_iv: "81dc44fb39c54966a85ebc8621399d0b"
server_key_version: "encryptionkey@0"
I20260618 23:30:04.406524 25925 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:30:04.407559 25940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:04.407922 25925 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:30:04.408049 25925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "edf58ae2793e418bb213d5748f8a238e"
format_stamp: "Formatted at 2026-06-18 23:30:04 on dist-test-slave-47qt"
server_key: "e0951b468aac14873f1b015de5fb73e9"
server_key_iv: "81dc44fb39c54966a85ebc8621399d0b"
server_key_version: "encryptionkey@0"
I20260618 23:30:04.408150 25925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:04.425323 25925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:04.428467 25925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:04.428705 25925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:04.432865 25925 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:37283
I20260618 23:30:04.432900 25992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:37283 every 8 connection(s)
I20260618 23:30:04.433290 25925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:30:04.434552 25993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:04.436383 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 25925
I20260618 23:30:04.436561 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:30:04.436753 8350 external_mini_cluster.cc:1428] Setting key cabf316ca0863ead15312b77cfd159c3
I20260618 23:30:04.438249 25993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edf58ae2793e418bb213d5748f8a238e: Bootstrap starting.
I20260618 23:30:04.439155 25993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edf58ae2793e418bb213d5748f8a238e: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:04.439551 25993 log.cc:824] T 00000000000000000000000000000000 P edf58ae2793e418bb213d5748f8a238e: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:04.440539 25993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edf58ae2793e418bb213d5748f8a238e: No bootstrap required, opened a new log
I20260618 23:30:04.443373 25993 raft_consensus.cc:348] T 00000000000000000000000000000000 P edf58ae2793e418bb213d5748f8a238e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf58ae2793e418bb213d5748f8a238e" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 37283 } }
I20260618 23:30:04.443602 25993 raft_consensus.cc:374] T 00000000000000000000000000000000 P edf58ae2793e418bb213d5748f8a238e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:04.444010 25993 raft_consensus.cc:729] T 00000000000000000000000000000000 P edf58ae2793e418bb213d5748f8a238e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edf58ae2793e418bb213d5748f8a238e, State: Initialized, Role: FOLLOWER
I20260618 23:30:04.444307 25993 consensus_queue.cc:260] T 00000000000000000000000000000000 P edf58ae2793e418bb213d5748f8a238e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf58ae2793e418bb213d5748f8a238e" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 37283 } }
I20260618 23:30:04.444500 25993 raft_consensus.cc:388] T 00000000000000000000000000000000 P edf58ae2793e418bb213d5748f8a238e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:30:04.444535 25993 raft_consensus.cc:482] T 00000000000000000000000000000000 P edf58ae2793e418bb213d5748f8a238e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:30:04.444631 25993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edf58ae2793e418bb213d5748f8a238e [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:04.445711 25993 raft_consensus.cc:504] T 00000000000000000000000000000000 P edf58ae2793e418bb213d5748f8a238e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf58ae2793e418bb213d5748f8a238e" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 37283 } }
I20260618 23:30:04.445874 25993 leader_election.cc:302] T 00000000000000000000000000000000 P edf58ae2793e418bb213d5748f8a238e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edf58ae2793e418bb213d5748f8a238e; no voters:
I20260618 23:30:04.446096 25993 leader_election.cc:288] T 00000000000000000000000000000000 P edf58ae2793e418bb213d5748f8a238e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:30:04.446201 25998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edf58ae2793e418bb213d5748f8a238e [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:30:04.446440 25993 sys_catalog.cc:564] T 00000000000000000000000000000000 P edf58ae2793e418bb213d5748f8a238e [sys.catalog]: configured and running, proceeding with master startup.
Jun 18 23:30:04 dist-test-slave-47qt krb5kdc[25912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:30:04.447168 25998 raft_consensus.cc:686] T 00000000000000000000000000000000 P edf58ae2793e418bb213d5748f8a238e [term 1 LEADER]: Becoming Leader. State: Replica: edf58ae2793e418bb213d5748f8a238e, State: Running, Role: LEADER
I20260618 23:30:04.447353 25998 consensus_queue.cc:237] T 00000000000000000000000000000000 P edf58ae2793e418bb213d5748f8a238e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf58ae2793e418bb213d5748f8a238e" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 37283 } }
I20260618 23:30:04.448132 25999 sys_catalog.cc:455] T 00000000000000000000000000000000 P edf58ae2793e418bb213d5748f8a238e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edf58ae2793e418bb213d5748f8a238e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf58ae2793e418bb213d5748f8a238e" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 37283 } } }
I20260618 23:30:04.448211 25999 sys_catalog.cc:458] T 00000000000000000000000000000000 P edf58ae2793e418bb213d5748f8a238e [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:04.447897 25998 sys_catalog.cc:455] T 00000000000000000000000000000000 P edf58ae2793e418bb213d5748f8a238e [sys.catalog]: SysCatalogTable state changed. Reason: New leader edf58ae2793e418bb213d5748f8a238e. Latest consensus state: current_term: 1 leader_uuid: "edf58ae2793e418bb213d5748f8a238e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edf58ae2793e418bb213d5748f8a238e" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 37283 } } }
I20260618 23:30:04.448710 25998 sys_catalog.cc:458] T 00000000000000000000000000000000 P edf58ae2793e418bb213d5748f8a238e [sys.catalog]: This master's current role is: LEADER
W20260618 23:30:04.448771 26012 catalog_manager.cc:1558] T 00000000000000000000000000000000 P edf58ae2793e418bb213d5748f8a238e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 23:30:04.448822 26012 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:30:04.448956 26013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:30:04.449177 26013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:30:04.450982 26013 catalog_manager.cc:1347] Generated new cluster ID: ef955438f88f4ce4a3df8259e7d07d6b
I20260618 23:30:04.451046 26013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:30:04.452795 25996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:04.437496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51654 (local address 127.8.39.190:37283)
0618 23:30:04.437680 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:04.437689 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:04.437719 (+ 30us) server_negotiation.cc:407] Connection header received
0618 23:30:04.438255 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:04.438263 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:04.438343 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:04.438508 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:04.439171 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:04.440859 (+ 1688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:04.441916 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:04.442753 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:04.442935 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:04.447162 (+ 4227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:04.447168 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:04.447168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:04.447185 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:04.450066 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:04.450636 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:04.450642 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:04.450642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:04.451089 (+ 447us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:04.451453 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:04.451456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:04.451457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:04.452021 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:04.452243 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:04.452390 (+ 147us) server_negotiation.cc:299] Negotiation successful
0618 23:30:04.452566 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":97,"threads_started":1}
I20260618 23:30:04.465194 26013 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:30:04.465864 26013 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:30:04.469511 26013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edf58ae2793e418bb213d5748f8a238e: Generated new TSK 0
I20260618 23:30:04.469789 26013 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.39.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
I20260618 23:30:04.509969 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.129:0
--local_ip_for_outbound_sockets=127.8.39.129
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:37283
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:45363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:30:04.618772 26019 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:04.619014 26019 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:04.619051 26019 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:04.620221 26019 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:04.620296 26019 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:04.620340 26019 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.129
I20260618 23:30:04.621914 26019 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:45363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:37283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26019
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.129
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:04.622179 26019 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:04.622421 26019 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:04.628366 26027 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:04.628374 26024 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:04.628371 26025 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:04.628734 26019 server_base.cc:1034] running on GCE node
I20260618 23:30:04.628989 26019 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:04.629307 26019 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:04.630489 26019 hybrid_clock.cc:648] HybridClock initialized: now 1781825404630482 us; error 37 us; skew 500 ppm
Jun 18 23:30:04 dist-test-slave-47qt krb5kdc[25912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825404, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:04.633033 26019 init.cc:377] Logged in from keytab as kudu/127.8.39.129@KRBTEST.COM (short username kudu)
I20260618 23:30:04.633487 26019 webserver.cc:460] Webserver started at http://127.8.39.129:38165/ using document root <none> and password file <none>
I20260618 23:30:04.633687 26019 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:04.633728 26019 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:04.633812 26019 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:04.634790 26019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-0/data/instance:
uuid: "b180823cfe9b432fbe633276d18862b8"
format_stamp: "Formatted at 2026-06-18 23:30:04 on dist-test-slave-47qt"
server_key: "bab4db234eda82c046da97c0eb7ab79e"
server_key_iv: "bea799b0615ed806178b8301ed19d809"
server_key_version: "encryptionkey@0"
I20260618 23:30:04.635082 26019 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance:
uuid: "b180823cfe9b432fbe633276d18862b8"
format_stamp: "Formatted at 2026-06-18 23:30:04 on dist-test-slave-47qt"
server_key: "bab4db234eda82c046da97c0eb7ab79e"
server_key_iv: "bea799b0615ed806178b8301ed19d809"
server_key_version: "encryptionkey@0"
I20260618 23:30:04.636343 26019 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:30:04.637162 26034 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:04.637338 26019 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260618 23:30:04.637410 26019 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-0/wal
uuid: "b180823cfe9b432fbe633276d18862b8"
format_stamp: "Formatted at 2026-06-18 23:30:04 on dist-test-slave-47qt"
server_key: "bab4db234eda82c046da97c0eb7ab79e"
server_key_iv: "bea799b0615ed806178b8301ed19d809"
server_key_version: "encryptionkey@0"
I20260618 23:30:04.637506 26019 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:04.666417 26019 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:04.668603 26019 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:04.668767 26019 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:04.669044 26019 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:30:04.669417 26019 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:30:04.669476 26019 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:04.669519 26019 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:30:04.669580 26019 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:04.675294 26019 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.129:40085
I20260618 23:30:04.675339 26147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.129:40085 every 8 connection(s)
I20260618 23:30:04.675658 26019 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
I20260618 23:30:04.676860 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 26019
I20260618 23:30:04.676944 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance
I20260618 23:30:04.677260 8350 external_mini_cluster.cc:1428] Setting key 909ef10964f0a8ea6cf0bdeac1509db4
Jun 18 23:30:04 dist-test-slave-47qt krb5kdc[25912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825404, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:30:04.686751 25996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:04.676424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:40867 (local address 127.8.39.190:37283)
0618 23:30:04.676517 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:04.676519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:04.676718 (+ 199us) server_negotiation.cc:407] Connection header received
0618 23:30:04.677476 (+ 758us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:04.677477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:04.677487 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:04.677567 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:04.678699 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:04.679195 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:04.680642 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:04.680861 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:04.680928 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:04.683901 (+ 2973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:04.683902 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:04.683903 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:04.683911 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:04.685329 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:04.685770 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:04.685771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:04.685771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:04.685831 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:04.686094 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:04.686095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:04.686096 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:04.686212 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:04.686290 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:04.686569 (+ 279us) server_negotiation.cc:299] Negotiation successful
0618 23:30:04.686658 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
WARNING: no policy specified for kudu/127.8.39.130@KRBTEST.COM; defaulting to no policy
I20260618 23:30:04.686954 26150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:04.676385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37283 (local address 127.8.39.129:40867)
0618 23:30:04.676570 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:04.676617 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:04.677357 (+ 740us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:04.677666 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:04.677670 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:04.678392 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:04.678590 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:04.678594 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:04.679293 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:04.679294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:04.680520 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:04.680522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:04.680989 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:04.680990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:04.681265 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:04.682016 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:30:04.682028 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:04.683777 (+ 1749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:04.685447 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:04.685453 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:04.685456 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:04.685678 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:04.685900 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:04.685901 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:04.685901 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:04.686010 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:04.686281 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:04.686287 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:04.686428 (+ 141us) client_negotiation.cc:769] Sending connection context
0618 23:30:04.686530 (+ 102us) client_negotiation.cc:240] Negotiation successful
0618 23:30:04.686694 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":87,"threads_started":1}
I20260618 23:30:04.687460 26148 heartbeater.cc:347] Connected to a master server at 127.8.39.190:37283
I20260618 23:30:04.687546 26148 heartbeater.cc:464] Registering TS with master...
I20260618 23:30:04.687786 26148 heartbeater.cc:511] Master 127.8.39.190:37283 requested a full tablet report, sending...
I20260618 23:30:04.688341 25957 ts_manager.cc:194] Registered new tserver with Master: b180823cfe9b432fbe633276d18862b8 (127.8.39.129:40085)
I20260618 23:30:04.689047 25957 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.129@KRBTEST.COM'} at 127.8.39.129:40867
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
I20260618 23:30:04.704458 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.130:0
--local_ip_for_outbound_sockets=127.8.39.130
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:37283
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:45363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:30:04.792188 26153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:04.792404 26153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:04.792434 26153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:04.793619 26153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:04.793658 26153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:04.793685 26153 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.130
I20260618 23:30:04.795099 26153 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:45363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:37283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.130
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:04.795302 26153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:04.795485 26153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:04.801344 26158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:04.801338 26161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:04.801338 26159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:04.801391 26153 server_base.cc:1034] running on GCE node
I20260618 23:30:04.801728 26153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:04.801978 26153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:04.803126 26153 hybrid_clock.cc:648] HybridClock initialized: now 1781825404803103 us; error 40 us; skew 500 ppm
Jun 18 23:30:04 dist-test-slave-47qt krb5kdc[25912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825404, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:04.805696 26153 init.cc:377] Logged in from keytab as kudu/127.8.39.130@KRBTEST.COM (short username kudu)
I20260618 23:30:04.806185 26153 webserver.cc:460] Webserver started at http://127.8.39.130:34665/ using document root <none> and password file <none>
I20260618 23:30:04.806439 26153 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:04.806509 26153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:04.806680 26153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:04.807753 26153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-1/data/instance:
uuid: "edbe4167df1a43eaa04439e4b02e8932"
format_stamp: "Formatted at 2026-06-18 23:30:04 on dist-test-slave-47qt"
server_key: "1f44bd9fc08eefadf814cde4f1d0ab79"
server_key_iv: "a1a8cdab73c194eef278bb51661b168d"
server_key_version: "encryptionkey@0"
I20260618 23:30:04.808048 26153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance:
uuid: "edbe4167df1a43eaa04439e4b02e8932"
format_stamp: "Formatted at 2026-06-18 23:30:04 on dist-test-slave-47qt"
server_key: "1f44bd9fc08eefadf814cde4f1d0ab79"
server_key_iv: "a1a8cdab73c194eef278bb51661b168d"
server_key_version: "encryptionkey@0"
I20260618 23:30:04.809453 26153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260618 23:30:04.810199 26168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:04.810448 26153 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:30:04.810530 26153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-1/wal
uuid: "edbe4167df1a43eaa04439e4b02e8932"
format_stamp: "Formatted at 2026-06-18 23:30:04 on dist-test-slave-47qt"
server_key: "1f44bd9fc08eefadf814cde4f1d0ab79"
server_key_iv: "a1a8cdab73c194eef278bb51661b168d"
server_key_version: "encryptionkey@0"
I20260618 23:30:04.810585 26153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:04.820276 26153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:04.822680 26153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:04.822794 26153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:04.823032 26153 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:30:04.823367 26153 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:30:04.823407 26153 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:04.823431 26153 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:30:04.823443 26153 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:04.830163 26153 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.130:42037
I20260618 23:30:04.830230 26281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.130:42037 every 8 connection(s)
I20260618 23:30:04.830592 26153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
Jun 18 23:30:04 dist-test-slave-47qt krb5kdc[25912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825404, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:30:04.839969 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 26153
I20260618 23:30:04.840085 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance
I20260618 23:30:04.840319 8350 external_mini_cluster.cc:1428] Setting key 356e97b5eaa4c587d23ee7cedbfa8153
I20260618 23:30:04.840911 25996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:04.831386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:34627 (local address 127.8.39.190:37283)
0618 23:30:04.831513 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:04.831516 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:04.831733 (+ 217us) server_negotiation.cc:407] Connection header received
0618 23:30:04.832308 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:04.832309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:04.832323 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:04.832393 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:04.833449 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:04.833962 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:04.835022 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:04.835237 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:04.835333 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:04.837564 (+ 2231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:04.837565 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:04.837566 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:04.837573 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:04.838931 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:04.839626 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:04.839627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:04.839627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:04.839683 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:04.840115 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:04.840116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:04.840116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:04.840261 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:04.840318 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:04.840730 (+ 412us) server_negotiation.cc:299] Negotiation successful
0618 23:30:04.840814 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260618 23:30:04.841176 26284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:04.831353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37283 (local address 127.8.39.130:34627)
0618 23:30:04.831601 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:04.831639 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:04.832180 (+ 541us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:04.832461 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:04.832466 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:04.833073 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:04.833346 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:04.833351 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:04.834093 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:04.834094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:04.834902 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:04.834904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:04.835344 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:04.835346 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:04.835482 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:04.836013 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:30:04.836021 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:04.837466 (+ 1445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:04.839085 (+ 1619us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:04.839099 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:04.839102 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:04.839454 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:04.839835 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:04.839836 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:04.839837 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:04.840015 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:04.840338 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:04.840345 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:04.840537 (+ 192us) client_negotiation.cc:769] Sending connection context
0618 23:30:04.840701 (+ 164us) client_negotiation.cc:240] Negotiation successful
0618 23:30:04.840893 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":100,"threads_started":1}
I20260618 23:30:04.841682 26282 heartbeater.cc:347] Connected to a master server at 127.8.39.190:37283
I20260618 23:30:04.841801 26282 heartbeater.cc:464] Registering TS with master...
I20260618 23:30:04.841976 26282 heartbeater.cc:511] Master 127.8.39.190:37283 requested a full tablet report, sending...
I20260618 23:30:04.842348 25957 ts_manager.cc:194] Registered new tserver with Master: edbe4167df1a43eaa04439e4b02e8932 (127.8.39.130:42037)
I20260618 23:30:04.842779 25957 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.130@KRBTEST.COM'} at 127.8.39.130:34627
WARNING: no policy specified for kudu/127.8.39.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
I20260618 23:30:04.866372 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.131:0
--local_ip_for_outbound_sockets=127.8.39.131
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:37283
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:45363
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:30:04.960086 26287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:04.960331 26287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:04.960359 26287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:04.961808 26287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:04.961859 26287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:04.961891 26287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.131
I20260618 23:30:04.963538 26287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:45363
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:37283
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.131
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:04.963825 26287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:04.964078 26287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:04.969192 26295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:04.969200 26293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:04.969200 26292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:04.969508 26287 server_base.cc:1034] running on GCE node
I20260618 23:30:04.969736 26287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:04.969940 26287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:04.971101 26287 hybrid_clock.cc:648] HybridClock initialized: now 1781825404971086 us; error 29 us; skew 500 ppm
Jun 18 23:30:04 dist-test-slave-47qt krb5kdc[25912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825404, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:04.973611 26287 init.cc:377] Logged in from keytab as kudu/127.8.39.131@KRBTEST.COM (short username kudu)
I20260618 23:30:04.974049 26287 webserver.cc:460] Webserver started at http://127.8.39.131:44779/ using document root <none> and password file <none>
I20260618 23:30:04.974296 26287 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:04.974342 26287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:04.974472 26287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:04.975610 26287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-2/data/instance:
uuid: "e279919f6857429f8330ac5c9916898b"
format_stamp: "Formatted at 2026-06-18 23:30:04 on dist-test-slave-47qt"
server_key: "701275fb11e3bc9e324e00fd75153ba8"
server_key_iv: "6eaf77ce069bc99874369e641e4fa1ca"
server_key_version: "encryptionkey@0"
I20260618 23:30:04.975966 26287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance:
uuid: "e279919f6857429f8330ac5c9916898b"
format_stamp: "Formatted at 2026-06-18 23:30:04 on dist-test-slave-47qt"
server_key: "701275fb11e3bc9e324e00fd75153ba8"
server_key_iv: "6eaf77ce069bc99874369e641e4fa1ca"
server_key_version: "encryptionkey@0"
I20260618 23:30:04.977628 26287 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:30:04.978493 26302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:04.978766 26287 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:30:04.978847 26287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-2/wal
uuid: "e279919f6857429f8330ac5c9916898b"
format_stamp: "Formatted at 2026-06-18 23:30:04 on dist-test-slave-47qt"
server_key: "701275fb11e3bc9e324e00fd75153ba8"
server_key_iv: "6eaf77ce069bc99874369e641e4fa1ca"
server_key_version: "encryptionkey@0"
I20260618 23:30:04.978946 26287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:04.999135 26287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:05.001534 26287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:05.001946 26287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:05.002226 26287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:30:05.002538 26287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:30:05.002604 26287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:05.002640 26287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:30:05.002668 26287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:05.009665 26287 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.131:40541
I20260618 23:30:05.009804 26415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.131:40541 every 8 connection(s)
I20260618 23:30:05.010098 26287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
I20260618 23:30:05.012228 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 26287
I20260618 23:30:05.012332 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance
I20260618 23:30:05.012532 8350 external_mini_cluster.cc:1428] Setting key 5a385fd13bc996b418642ad75f3f1182
Jun 18 23:30:05 dist-test-slave-47qt krb5kdc[25912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825404, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:30:05.020294 25996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:05.011018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:52557 (local address 127.8.39.190:37283)
0618 23:30:05.011105 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:05.011107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:05.011223 (+ 116us) server_negotiation.cc:407] Connection header received
0618 23:30:05.011775 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:05.011776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:05.011788 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:05.011847 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:05.013006 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.013686 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.014743 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.014951 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.015037 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:05.017267 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:05.017269 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:05.017269 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:05.017276 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:05.018736 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:05.019219 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:05.019220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:05.019221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:05.019289 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:05.019575 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:05.019576 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:05.019577 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:05.019716 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:05.019800 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:05.020096 (+ 296us) server_negotiation.cc:299] Negotiation successful
0618 23:30:05.020184 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260618 23:30:05.020545 26418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:05.010920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:37283 (local address 127.8.39.131:52557)
0618 23:30:05.011081 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:05.011133 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:05.011666 (+ 533us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:05.011925 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:05.011928 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:05.012669 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:05.012902 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:05.012908 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.013778 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.013780 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:05.014645 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:05.014647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.015045 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.015046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:05.015179 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:05.015691 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:30:05.015699 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:05.017104 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:05.018856 (+ 1752us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:05.018863 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:05.018866 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:05.019109 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:05.019364 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:05.019365 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:05.019365 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:05.019487 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:05.019790 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:05.019796 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:05.019951 (+ 155us) client_negotiation.cc:769] Sending connection context
0618 23:30:05.020074 (+ 123us) client_negotiation.cc:240] Negotiation successful
0618 23:30:05.020264 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260618 23:30:05.021651 26416 heartbeater.cc:347] Connected to a master server at 127.8.39.190:37283
I20260618 23:30:05.021750 26416 heartbeater.cc:464] Registering TS with master...
I20260618 23:30:05.021893 26416 heartbeater.cc:511] Master 127.8.39.190:37283 requested a full tablet report, sending...
I20260618 23:30:05.022408 25957 ts_manager.cc:194] Registered new tserver with Master: e279919f6857429f8330ac5c9916898b (127.8.39.131:40541)
I20260618 23:30:05.023037 25957 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.131@KRBTEST.COM'} at 127.8.39.131:52557
I20260618 23:30:05.025382 8350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:30:05.034346 25996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:05.026883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51664 (local address 127.8.39.190:37283)
0618 23:30:05.026955 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:05.026957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:05.026964 (+ 7us) server_negotiation.cc:407] Connection header received
0618 23:30:05.026985 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:05.026986 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:05.026995 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:05.027069 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:05.027633 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.028119 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.028852 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.029062 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.029157 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:05.030973 (+ 1816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:05.030975 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:05.030976 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:05.030984 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:05.032553 (+ 1569us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:05.033096 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:05.033097 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:05.033098 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:05.033203 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:05.033616 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:05.033618 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:05.033618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:05.033812 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:05.033989 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:05.034144 (+ 155us) server_negotiation.cc:299] Negotiation successful
0618 23:30:05.034214 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260618 23:30:05.036147 25957 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51664:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:30:05.037925 25957 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:30:05.046105 26431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:05.041609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56862 (local address 127.8.39.129:40085)
0618 23:30:05.042030 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:05.042035 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:05.042045 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:30:05.042085 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:05.042087 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:05.042102 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:05.042196 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:30:05.042712 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.043472 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.045034 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.045629 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.045741 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:05.045808 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:05.045968 (+ 160us) server_negotiation.cc:299] Negotiation successful
0618 23:30:05.046009 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":127,"threads_started":1}
I20260618 23:30:05.046133 26429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:05.041394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:40085 (local address 127.0.0.1:56862)
0618 23:30:05.041836 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:05.041847 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:05.042017 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:05.042585 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:05.042586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:05.042589 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:30:05.042634 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:05.042635 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.043588 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.043590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:05.044932 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:05.044935 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.045744 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.045745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:05.045858 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:05.045873 (+ 15us) client_negotiation.cc:769] Sending connection context
0618 23:30:05.045950 (+ 77us) client_negotiation.cc:240] Negotiation successful
0618 23:30:05.046006 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":421,"mutex_wait_us":1,"thread_start_us":74,"threads_started":1}
I20260618 23:30:05.046152 26427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:05.041371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:40541 (local address 127.0.0.1:58824)
0618 23:30:05.041621 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:05.041664 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:05.041737 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:05.042575 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:05.042577 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:05.042581 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:30:05.042673 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:05.042676 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.044219 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.044220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:05.044950 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:05.044953 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.045905 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.045906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:05.045968 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:05.045972 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:30:05.046003 (+ 31us) client_negotiation.cc:240] Negotiation successful
0618 23:30:05.046045 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":1,"thread_start_us":62,"threads_started":1}
I20260618 23:30:05.046756 26430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:05.041536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58824 (local address 127.8.39.131:40541)
0618 23:30:05.042348 (+ 812us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:05.042353 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:05.042361 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:30:05.042396 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:05.042398 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:05.042412 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:05.042510 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:30:05.042798 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.044092 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.045076 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.045784 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.046509 (+ 725us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:05.046576 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:05.046617 (+ 41us) server_negotiation.cc:299] Negotiation successful
0618 23:30:05.046669 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":777,"thread_start_us":105,"threads_started":1}
I20260618 23:30:05.046969 26428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:05.041520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:42037 (local address 127.0.0.1:43864)
0618 23:30:05.041862 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:05.041871 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:05.041908 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:05.042357 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:05.042360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:05.042362 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:30:05.042450 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:05.042453 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.044201 (+ 1748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.044202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:05.045330 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:05.045332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.046714 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.046715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:05.046808 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:05.046812 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:30:05.046845 (+ 33us) client_negotiation.cc:240] Negotiation successful
0618 23:30:05.046893 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":325,"mutex_wait_us":28,"thread_start_us":113,"threads_started":1}
I20260618 23:30:05.047392 26432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:05.041818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43864 (local address 127.8.39.130:42037)
0618 23:30:05.041980 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:05.041985 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:05.041997 (+ 12us) server_negotiation.cc:407] Connection header received
0618 23:30:05.042163 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:05.042169 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:05.042195 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:05.042314 (+ 119us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:30:05.042572 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.044062 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.045420 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.046143 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.047178 (+ 1035us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:05.047241 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:05.047266 (+ 25us) server_negotiation.cc:299] Negotiation successful
0618 23:30:05.047310 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":97,"threads_started":1}
I20260618 23:30:05.047576 26082 tablet_service.cc:1467] Processing CreateTablet for tablet dedd7ef0c54c41a7881525f361f79d53 (DEFAULT_TABLE table=test-table [id=2fdf67348c7745cab2df48f9a4b5b9ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:30:05.047976 26082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dedd7ef0c54c41a7881525f361f79d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:05.047997 26216 tablet_service.cc:1467] Processing CreateTablet for tablet dedd7ef0c54c41a7881525f361f79d53 (DEFAULT_TABLE table=test-table [id=2fdf67348c7745cab2df48f9a4b5b9ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:30:05.048363 26216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dedd7ef0c54c41a7881525f361f79d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:05.049624 26350 tablet_service.cc:1467] Processing CreateTablet for tablet dedd7ef0c54c41a7881525f361f79d53 (DEFAULT_TABLE table=test-table [id=2fdf67348c7745cab2df48f9a4b5b9ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:30:05.049949 26350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dedd7ef0c54c41a7881525f361f79d53. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:05.051011 26433 tablet_bootstrap.cc:492] T dedd7ef0c54c41a7881525f361f79d53 P b180823cfe9b432fbe633276d18862b8: Bootstrap starting.
I20260618 23:30:05.051501 26434 tablet_bootstrap.cc:492] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932: Bootstrap starting.
I20260618 23:30:05.051847 26433 tablet_bootstrap.cc:654] T dedd7ef0c54c41a7881525f361f79d53 P b180823cfe9b432fbe633276d18862b8: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:05.052172 26434 tablet_bootstrap.cc:654] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:05.052474 26433 log.cc:824] T dedd7ef0c54c41a7881525f361f79d53 P b180823cfe9b432fbe633276d18862b8: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:05.052762 26434 log.cc:824] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:05.053278 26437 tablet_bootstrap.cc:492] T dedd7ef0c54c41a7881525f361f79d53 P e279919f6857429f8330ac5c9916898b: Bootstrap starting.
I20260618 23:30:05.053378 26433 tablet_bootstrap.cc:492] T dedd7ef0c54c41a7881525f361f79d53 P b180823cfe9b432fbe633276d18862b8: No bootstrap required, opened a new log
I20260618 23:30:05.053452 26433 ts_tablet_manager.cc:1397] T dedd7ef0c54c41a7881525f361f79d53 P b180823cfe9b432fbe633276d18862b8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260618 23:30:05.053478 26434 tablet_bootstrap.cc:492] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932: No bootstrap required, opened a new log
I20260618 23:30:05.053534 26434 ts_tablet_manager.cc:1397] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260618 23:30:05.053867 26437 tablet_bootstrap.cc:654] T dedd7ef0c54c41a7881525f361f79d53 P e279919f6857429f8330ac5c9916898b: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:05.054355 26437 log.cc:824] T dedd7ef0c54c41a7881525f361f79d53 P e279919f6857429f8330ac5c9916898b: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:05.055019 26437 tablet_bootstrap.cc:492] T dedd7ef0c54c41a7881525f361f79d53 P e279919f6857429f8330ac5c9916898b: No bootstrap required, opened a new log
I20260618 23:30:05.055111 26437 ts_tablet_manager.cc:1397] T dedd7ef0c54c41a7881525f361f79d53 P e279919f6857429f8330ac5c9916898b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:30:05.055481 26434 raft_consensus.cc:348] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e279919f6857429f8330ac5c9916898b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 40541 } } peers { permanent_uuid: "edbe4167df1a43eaa04439e4b02e8932" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42037 } } peers { permanent_uuid: "b180823cfe9b432fbe633276d18862b8" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40085 } }
I20260618 23:30:05.055487 26433 raft_consensus.cc:348] T dedd7ef0c54c41a7881525f361f79d53 P b180823cfe9b432fbe633276d18862b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e279919f6857429f8330ac5c9916898b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 40541 } } peers { permanent_uuid: "edbe4167df1a43eaa04439e4b02e8932" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42037 } } peers { permanent_uuid: "b180823cfe9b432fbe633276d18862b8" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40085 } }
I20260618 23:30:05.055805 26433 raft_consensus.cc:374] T dedd7ef0c54c41a7881525f361f79d53 P b180823cfe9b432fbe633276d18862b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:05.055805 26434 raft_consensus.cc:374] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:05.055927 26433 raft_consensus.cc:729] T dedd7ef0c54c41a7881525f361f79d53 P b180823cfe9b432fbe633276d18862b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b180823cfe9b432fbe633276d18862b8, State: Initialized, Role: FOLLOWER
I20260618 23:30:05.055927 26434 raft_consensus.cc:729] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edbe4167df1a43eaa04439e4b02e8932, State: Initialized, Role: FOLLOWER
I20260618 23:30:05.056115 26433 consensus_queue.cc:260] T dedd7ef0c54c41a7881525f361f79d53 P b180823cfe9b432fbe633276d18862b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e279919f6857429f8330ac5c9916898b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 40541 } } peers { permanent_uuid: "edbe4167df1a43eaa04439e4b02e8932" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42037 } } peers { permanent_uuid: "b180823cfe9b432fbe633276d18862b8" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40085 } }
I20260618 23:30:05.056115 26434 consensus_queue.cc:260] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e279919f6857429f8330ac5c9916898b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 40541 } } peers { permanent_uuid: "edbe4167df1a43eaa04439e4b02e8932" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42037 } } peers { permanent_uuid: "b180823cfe9b432fbe633276d18862b8" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40085 } }
I20260618 23:30:05.056625 26433 ts_tablet_manager.cc:1428] T dedd7ef0c54c41a7881525f361f79d53 P b180823cfe9b432fbe633276d18862b8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:30:05.056680 26282 heartbeater.cc:503] Master 127.8.39.190:37283 was elected leader, sending a full tablet report...
I20260618 23:30:05.056808 26148 heartbeater.cc:503] Master 127.8.39.190:37283 was elected leader, sending a full tablet report...
I20260618 23:30:05.056944 26434 ts_tablet_manager.cc:1428] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:30:05.057291 26437 raft_consensus.cc:348] T dedd7ef0c54c41a7881525f361f79d53 P e279919f6857429f8330ac5c9916898b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e279919f6857429f8330ac5c9916898b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 40541 } } peers { permanent_uuid: "edbe4167df1a43eaa04439e4b02e8932" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42037 } } peers { permanent_uuid: "b180823cfe9b432fbe633276d18862b8" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40085 } }
I20260618 23:30:05.057490 26437 raft_consensus.cc:374] T dedd7ef0c54c41a7881525f361f79d53 P e279919f6857429f8330ac5c9916898b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:05.057749 26437 raft_consensus.cc:729] T dedd7ef0c54c41a7881525f361f79d53 P e279919f6857429f8330ac5c9916898b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e279919f6857429f8330ac5c9916898b, State: Initialized, Role: FOLLOWER
I20260618 23:30:05.057881 26437 consensus_queue.cc:260] T dedd7ef0c54c41a7881525f361f79d53 P e279919f6857429f8330ac5c9916898b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e279919f6857429f8330ac5c9916898b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 40541 } } peers { permanent_uuid: "edbe4167df1a43eaa04439e4b02e8932" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42037 } } peers { permanent_uuid: "b180823cfe9b432fbe633276d18862b8" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40085 } }
I20260618 23:30:05.058151 26437 ts_tablet_manager.cc:1428] T dedd7ef0c54c41a7881525f361f79d53 P e279919f6857429f8330ac5c9916898b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:30:05.058228 26416 heartbeater.cc:503] Master 127.8.39.190:37283 was elected leader, sending a full tablet report...
W20260618 23:30:05.081439 26283 tablet.cc:2369] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 23:30:05.176473 26149 tablet.cc:2369] T dedd7ef0c54c41a7881525f361f79d53 P b180823cfe9b432fbe633276d18862b8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:30:05.218000 26439 raft_consensus.cc:482] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:30:05.218130 26439 raft_consensus.cc:504] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e279919f6857429f8330ac5c9916898b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 40541 } } peers { permanent_uuid: "edbe4167df1a43eaa04439e4b02e8932" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42037 } } peers { permanent_uuid: "b180823cfe9b432fbe633276d18862b8" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40085 } }
I20260618 23:30:05.218600 26439 leader_election.cc:288] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e279919f6857429f8330ac5c9916898b (127.8.39.131:40541), b180823cfe9b432fbe633276d18862b8 (127.8.39.129:40085)
I20260618 23:30:05.221482 26284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:05.218840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:40085 (local address 127.8.39.130:36651)
0618 23:30:05.218904 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:05.218915 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:05.218964 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:05.219142 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:05.219143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:05.219144 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:30:05.219224 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:05.219225 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.219872 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.219873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:05.220624 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:05.220626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.221203 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.221204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:05.221287 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:05.221290 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:30:05.221365 (+ 75us) client_negotiation.cc:240] Negotiation successful
0618 23:30:05.221408 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":3}
I20260618 23:30:05.221479 26431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:05.218956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:36651 (local address 127.8.39.129:40085)
0618 23:30:05.219011 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:05.219013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:05.219021 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:30:05.219050 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:05.219051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:05.219070 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:05.219138 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:30:05.219305 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.219784 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.220719 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.221081 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.221203 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:05.221212 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:05.221374 (+ 162us) server_negotiation.cc:299] Negotiation successful
0618 23:30:05.221410 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260618 23:30:05.221598 26430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:05.218936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:38569 (local address 127.8.39.131:40541)
0618 23:30:05.218990 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:05.218992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:05.219120 (+ 128us) server_negotiation.cc:407] Connection header received
0618 23:30:05.219189 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:05.219190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:05.219199 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:05.219273 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:30:05.219437 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.219893 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.220821 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.221249 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.221334 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:05.221342 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:05.221502 (+ 160us) server_negotiation.cc:299] Negotiation successful
0618 23:30:05.221537 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260618 23:30:05.221606 26442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:05.218858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:40541 (local address 127.8.39.130:38569)
0618 23:30:05.219033 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:05.219051 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:05.219130 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:05.219275 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:05.219276 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:05.219277 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:30:05.219356 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:05.219357 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.219979 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.219980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:05.220728 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:05.220730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.221335 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.221335 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:05.221416 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:05.221419 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:30:05.221483 (+ 64us) client_negotiation.cc:240] Negotiation successful
0618 23:30:05.221523 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":96,"threads_started":1}
I20260618 23:30:05.221835 26102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dedd7ef0c54c41a7881525f361f79d53" candidate_uuid: "edbe4167df1a43eaa04439e4b02e8932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b180823cfe9b432fbe633276d18862b8" is_pre_election: true
I20260618 23:30:05.221985 26102 raft_consensus.cc:2454] T dedd7ef0c54c41a7881525f361f79d53 P b180823cfe9b432fbe633276d18862b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edbe4167df1a43eaa04439e4b02e8932 in term 0.
I20260618 23:30:05.222194 26169 leader_election.cc:302] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b180823cfe9b432fbe633276d18862b8, edbe4167df1a43eaa04439e4b02e8932; no voters:
I20260618 23:30:05.222190 26370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dedd7ef0c54c41a7881525f361f79d53" candidate_uuid: "edbe4167df1a43eaa04439e4b02e8932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e279919f6857429f8330ac5c9916898b" is_pre_election: true
I20260618 23:30:05.222368 26370 raft_consensus.cc:2454] T dedd7ef0c54c41a7881525f361f79d53 P e279919f6857429f8330ac5c9916898b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edbe4167df1a43eaa04439e4b02e8932 in term 0.
I20260618 23:30:05.222385 26439 raft_consensus.cc:2781] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:30:05.222465 26439 raft_consensus.cc:482] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:30:05.222497 26439 raft_consensus.cc:3037] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:05.223246 26439 raft_consensus.cc:504] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e279919f6857429f8330ac5c9916898b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 40541 } } peers { permanent_uuid: "edbe4167df1a43eaa04439e4b02e8932" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42037 } } peers { permanent_uuid: "b180823cfe9b432fbe633276d18862b8" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40085 } }
I20260618 23:30:05.223410 26439 leader_election.cc:288] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932 [CANDIDATE]: Term 1 election: Requested vote from peers e279919f6857429f8330ac5c9916898b (127.8.39.131:40541), b180823cfe9b432fbe633276d18862b8 (127.8.39.129:40085)
I20260618 23:30:05.223584 26102 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dedd7ef0c54c41a7881525f361f79d53" candidate_uuid: "edbe4167df1a43eaa04439e4b02e8932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b180823cfe9b432fbe633276d18862b8"
I20260618 23:30:05.223867 26102 raft_consensus.cc:3037] T dedd7ef0c54c41a7881525f361f79d53 P b180823cfe9b432fbe633276d18862b8 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:05.223609 26370 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dedd7ef0c54c41a7881525f361f79d53" candidate_uuid: "edbe4167df1a43eaa04439e4b02e8932" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e279919f6857429f8330ac5c9916898b"
I20260618 23:30:05.224053 26370 raft_consensus.cc:3037] T dedd7ef0c54c41a7881525f361f79d53 P e279919f6857429f8330ac5c9916898b [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:05.224701 26102 raft_consensus.cc:2454] T dedd7ef0c54c41a7881525f361f79d53 P b180823cfe9b432fbe633276d18862b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edbe4167df1a43eaa04439e4b02e8932 in term 1.
I20260618 23:30:05.224951 26169 leader_election.cc:302] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b180823cfe9b432fbe633276d18862b8, edbe4167df1a43eaa04439e4b02e8932; no voters:
I20260618 23:30:05.225046 26439 raft_consensus.cc:2781] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:30:05.225203 26439 raft_consensus.cc:686] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932 [term 1 LEADER]: Becoming Leader. State: Replica: edbe4167df1a43eaa04439e4b02e8932, State: Running, Role: LEADER
I20260618 23:30:05.225427 26439 consensus_queue.cc:237] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e279919f6857429f8330ac5c9916898b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 40541 } } peers { permanent_uuid: "edbe4167df1a43eaa04439e4b02e8932" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42037 } } peers { permanent_uuid: "b180823cfe9b432fbe633276d18862b8" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40085 } }
I20260618 23:30:05.225600 26370 raft_consensus.cc:2454] T dedd7ef0c54c41a7881525f361f79d53 P e279919f6857429f8330ac5c9916898b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edbe4167df1a43eaa04439e4b02e8932 in term 1.
I20260618 23:30:05.226438 25957 catalog_manager.cc:5495] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932 reported cstate change: term changed from 0 to 1, leader changed from <none> to edbe4167df1a43eaa04439e4b02e8932 (127.8.39.130). New cstate: current_term: 1 leader_uuid: "edbe4167df1a43eaa04439e4b02e8932" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e279919f6857429f8330ac5c9916898b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 40541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edbe4167df1a43eaa04439e4b02e8932" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42037 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b180823cfe9b432fbe633276d18862b8" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40085 } health_report { overall_health: UNKNOWN } } }
W20260618 23:30:05.260787 26417 tablet.cc:2369] T dedd7ef0c54c41a7881525f361f79d53 P e279919f6857429f8330ac5c9916898b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:30:05.273190 26432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:05.270618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43880 (local address 127.8.39.130:42037)
0618 23:30:05.270711 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:05.270713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:05.270730 (+ 17us) server_negotiation.cc:407] Connection header received
0618 23:30:05.270764 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:05.270765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:05.270787 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:05.270847 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:30:05.270972 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.271443 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.272178 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.272374 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.272430 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:05.272574 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:30:05.272658 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:30:05.272741 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:05.273023 (+ 282us) server_negotiation.cc:299] Negotiation successful
0618 23:30:05.273037 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260618 23:30:05.275015 26102 raft_consensus.cc:1261] T dedd7ef0c54c41a7881525f361f79d53 P b180823cfe9b432fbe633276d18862b8 [term 1 FOLLOWER]: Refusing update from remote peer edbe4167df1a43eaa04439e4b02e8932: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:30:05.275014 26370 raft_consensus.cc:1261] T dedd7ef0c54c41a7881525f361f79d53 P e279919f6857429f8330ac5c9916898b [term 1 FOLLOWER]: Refusing update from remote peer edbe4167df1a43eaa04439e4b02e8932: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:30:05.275365 26439 consensus_queue.cc:1035] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b180823cfe9b432fbe633276d18862b8" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 40085 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:30:05.275501 26443 consensus_queue.cc:1035] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e279919f6857429f8330ac5c9916898b" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 40541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:30:05.278016 26448 mvcc.cc:205] Tried to move back new op lower bound from 7298356860003336192 to 7298356859803828224. Current Snapshot: MvccSnapshot[applied={T|T < 7298356860003336192}]
I20260618 23:30:05.280684 25955 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:30:05.280814 25955 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51664:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:30:05.281680 25955 catalog_manager.cc:5782] T 00000000000000000000000000000000 P edf58ae2793e418bb213d5748f8a238e: Sending DeleteTablet for 3 replicas of tablet dedd7ef0c54c41a7881525f361f79d53
I20260618 23:30:05.282058 26216 tablet_service.cc:1514] Processing DeleteTablet for tablet dedd7ef0c54c41a7881525f361f79d53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:30:05 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:43864
I20260618 23:30:05.282056 26350 tablet_service.cc:1514] Processing DeleteTablet for tablet dedd7ef0c54c41a7881525f361f79d53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:30:05 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:58824
I20260618 23:30:05.282341 26456 tablet_replica.cc:330] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932: stopping tablet replica
I20260618 23:30:05.282404 26082 tablet_service.cc:1514] Processing DeleteTablet for tablet dedd7ef0c54c41a7881525f361f79d53 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:30:05 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:56862
I20260618 23:30:05.282532 26456 raft_consensus.cc:2229] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932 [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:30:05.282754 26456 raft_consensus.cc:2258] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:30:05.282768 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 26019
I20260618 23:30:05.283277 26456 ts_tablet_manager.cc:1905] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:30:05.285084 26456 ts_tablet_manager.cc:1918] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:30:05.285264 26456 log.cc:1196] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-1/wal/wals/dedd7ef0c54c41a7881525f361f79d53
I20260618 23:30:05.285609 26456 ts_tablet_manager.cc:1939] T dedd7ef0c54c41a7881525f361f79d53 P edbe4167df1a43eaa04439e4b02e8932: Deleting consensus metadata
I20260618 23:30:05.286108 25943 catalog_manager.cc:4841] TS edbe4167df1a43eaa04439e4b02e8932 (127.8.39.130:42037): tablet dedd7ef0c54c41a7881525f361f79d53 (table test-table [id=2fdf67348c7745cab2df48f9a4b5b9ea]) successfully deleted
I20260618 23:30:05.286159 26457 tablet_replica.cc:330] T dedd7ef0c54c41a7881525f361f79d53 P e279919f6857429f8330ac5c9916898b: stopping tablet replica
I20260618 23:30:05.286386 26457 raft_consensus.cc:2229] T dedd7ef0c54c41a7881525f361f79d53 P e279919f6857429f8330ac5c9916898b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:30:05.286545 26457 raft_consensus.cc:2258] T dedd7ef0c54c41a7881525f361f79d53 P e279919f6857429f8330ac5c9916898b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:30:05.287328 26457 ts_tablet_manager.cc:1905] T dedd7ef0c54c41a7881525f361f79d53 P e279919f6857429f8330ac5c9916898b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:30:05.289376 26457 ts_tablet_manager.cc:1918] T dedd7ef0c54c41a7881525f361f79d53 P e279919f6857429f8330ac5c9916898b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:30:05.289449 26457 log.cc:1196] T dedd7ef0c54c41a7881525f361f79d53 P e279919f6857429f8330ac5c9916898b: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781825053542692-8350-0/minicluster-data/ts-2/wal/wals/dedd7ef0c54c41a7881525f361f79d53
I20260618 23:30:05.289726 26457 ts_tablet_manager.cc:1939] T dedd7ef0c54c41a7881525f361f79d53 P e279919f6857429f8330ac5c9916898b: Deleting consensus metadata
I20260618 23:30:05.290156 25942 catalog_manager.cc:4841] TS e279919f6857429f8330ac5c9916898b (127.8.39.131:40541): tablet dedd7ef0c54c41a7881525f361f79d53 (table test-table [id=2fdf67348c7745cab2df48f9a4b5b9ea]) successfully deleted
W20260618 23:30:05.291045 25941 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.39.129:40085 (error 108)
I20260618 23:30:05.291244 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 26153
I20260618 23:30:05.292250 26428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:05.292119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:40085 (local address 127.0.0.1:56872)
0618 23:30:05.292180 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:05.292224 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.129:40085: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260618 23:30:05.292441 25941 catalog_manager.cc:4584] TS b180823cfe9b432fbe633276d18862b8 (127.8.39.129:40085): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dedd7ef0c54c41a7881525f361f79d53: Network error: Client connection negotiation failed: client connection to 127.8.39.129:40085: connect: Connection refused (error 111)
I20260618 23:30:05.296072 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 26287
I20260618 23:30:05.303017 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 25925
2026-06-18T23:30:05Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2224 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-18T23:30:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:30:05Z Disabled control of system clock
I20260618 23:30:05.332134 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:41697
--webserver_interface=127.8.39.190
--webserver_port=0
--builtin_ntp_servers=127.8.39.148:39455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:41697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 23:30:05.423972 26463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:05.424198 26463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:05.424220 26463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:05.425550 26463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:30:05.425609 26463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:05.425622 26463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:05.425635 26463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:30:05.425648 26463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:30:05.427096 26463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:39455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:41697
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:41697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26463
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:05.427368 26463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:05.427676 26463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:05.432650 26468 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:05.432662 26469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:05.432673 26471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:05.432930 26463 server_base.cc:1034] running on GCE node
I20260618 23:30:05.433233 26463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:05.433490 26463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:05.434736 26463 hybrid_clock.cc:648] HybridClock initialized: now 1781825405434706 us; error 28 us; skew 500 ppm
I20260618 23:30:05.439769 26463 webserver.cc:460] Webserver started at http://127.8.39.190:38403/ using document root <none> and password file <none>
I20260618 23:30:05.440094 26463 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:05.440147 26463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:05.440296 26463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:05.441514 26463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "b20db0448a674518a2bbc8147e2422f0"
format_stamp: "Formatted at 2026-06-18 23:30:05 on dist-test-slave-47qt"
server_key: "7ae9f9071727b9a93c3c94a32e9096d5"
server_key_iv: "9a2768a25212581bf8cfc9d53e9604aa"
server_key_version: "encryptionkey@0"
I20260618 23:30:05.441895 26463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "b20db0448a674518a2bbc8147e2422f0"
format_stamp: "Formatted at 2026-06-18 23:30:05 on dist-test-slave-47qt"
server_key: "7ae9f9071727b9a93c3c94a32e9096d5"
server_key_iv: "9a2768a25212581bf8cfc9d53e9604aa"
server_key_version: "encryptionkey@0"
I20260618 23:30:05.443253 26463 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260618 23:30:05.444128 26477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:05.444454 26463 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:30:05.444581 26463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "b20db0448a674518a2bbc8147e2422f0"
format_stamp: "Formatted at 2026-06-18 23:30:05 on dist-test-slave-47qt"
server_key: "7ae9f9071727b9a93c3c94a32e9096d5"
server_key_iv: "9a2768a25212581bf8cfc9d53e9604aa"
server_key_version: "encryptionkey@0"
I20260618 23:30:05.444667 26463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:05.456705 26463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:05.457082 26463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:05.457275 26463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:05.461623 26463 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:41697
I20260618 23:30:05.461701 26529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:41697 every 8 connection(s)
I20260618 23:30:05.462141 26463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:30:05.463119 26530 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:05.465508 26530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b20db0448a674518a2bbc8147e2422f0: Bootstrap starting.
I20260618 23:30:05.466037 26530 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b20db0448a674518a2bbc8147e2422f0: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:05.466276 26530 log.cc:824] T 00000000000000000000000000000000 P b20db0448a674518a2bbc8147e2422f0: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:05.467255 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 26463
I20260618 23:30:05.467356 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:30:05.467377 26530 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b20db0448a674518a2bbc8147e2422f0: No bootstrap required, opened a new log
I20260618 23:30:05.467577 8350 external_mini_cluster.cc:1428] Setting key 50c3d32d3d0d93831616be8904babcff
I20260618 23:30:05.470844 26530 raft_consensus.cc:348] T 00000000000000000000000000000000 P b20db0448a674518a2bbc8147e2422f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b20db0448a674518a2bbc8147e2422f0" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41697 } }
I20260618 23:30:05.471035 26530 raft_consensus.cc:374] T 00000000000000000000000000000000 P b20db0448a674518a2bbc8147e2422f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:05.471066 26530 raft_consensus.cc:729] T 00000000000000000000000000000000 P b20db0448a674518a2bbc8147e2422f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b20db0448a674518a2bbc8147e2422f0, State: Initialized, Role: FOLLOWER
I20260618 23:30:05.471222 26530 consensus_queue.cc:260] T 00000000000000000000000000000000 P b20db0448a674518a2bbc8147e2422f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b20db0448a674518a2bbc8147e2422f0" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41697 } }
I20260618 23:30:05.471307 26530 raft_consensus.cc:388] T 00000000000000000000000000000000 P b20db0448a674518a2bbc8147e2422f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:30:05.471342 26530 raft_consensus.cc:482] T 00000000000000000000000000000000 P b20db0448a674518a2bbc8147e2422f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:30:05.471390 26530 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b20db0448a674518a2bbc8147e2422f0 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:05.471995 26530 raft_consensus.cc:504] T 00000000000000000000000000000000 P b20db0448a674518a2bbc8147e2422f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b20db0448a674518a2bbc8147e2422f0" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41697 } }
I20260618 23:30:05.472141 26530 leader_election.cc:302] T 00000000000000000000000000000000 P b20db0448a674518a2bbc8147e2422f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b20db0448a674518a2bbc8147e2422f0; no voters:
I20260618 23:30:05.472316 26530 leader_election.cc:288] T 00000000000000000000000000000000 P b20db0448a674518a2bbc8147e2422f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:30:05.472394 26535 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b20db0448a674518a2bbc8147e2422f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:30:05.472573 26530 sys_catalog.cc:564] T 00000000000000000000000000000000 P b20db0448a674518a2bbc8147e2422f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:30:05.472594 26535 raft_consensus.cc:686] T 00000000000000000000000000000000 P b20db0448a674518a2bbc8147e2422f0 [term 1 LEADER]: Becoming Leader. State: Replica: b20db0448a674518a2bbc8147e2422f0, State: Running, Role: LEADER
I20260618 23:30:05.472679 26535 consensus_queue.cc:237] T 00000000000000000000000000000000 P b20db0448a674518a2bbc8147e2422f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b20db0448a674518a2bbc8147e2422f0" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41697 } }
I20260618 23:30:05.473008 26535 sys_catalog.cc:455] T 00000000000000000000000000000000 P b20db0448a674518a2bbc8147e2422f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b20db0448a674518a2bbc8147e2422f0. Latest consensus state: current_term: 1 leader_uuid: "b20db0448a674518a2bbc8147e2422f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b20db0448a674518a2bbc8147e2422f0" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41697 } } }
I20260618 23:30:05.473107 26535 sys_catalog.cc:458] T 00000000000000000000000000000000 P b20db0448a674518a2bbc8147e2422f0 [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:05.473019 26536 sys_catalog.cc:455] T 00000000000000000000000000000000 P b20db0448a674518a2bbc8147e2422f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b20db0448a674518a2bbc8147e2422f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b20db0448a674518a2bbc8147e2422f0" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 41697 } } }
I20260618 23:30:05.473321 26536 sys_catalog.cc:458] T 00000000000000000000000000000000 P b20db0448a674518a2bbc8147e2422f0 [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:05.474030 26542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:30:05.474578 26542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:30:05.475538 26534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:05.468277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52780 (local address 127.8.39.190:41697)
0618 23:30:05.468518 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:05.468530 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:05.468574 (+ 44us) server_negotiation.cc:407] Connection header received
0618 23:30:05.469280 (+ 706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:05.469289 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:05.469367 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:05.469526 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:05.469776 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.471241 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.472068 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.472695 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.474589 (+ 1894us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:05.474825 (+ 236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:05.474829 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:05.474829 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:05.475002 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:05.475031 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:05.475041 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:05.475134 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:05.475227 (+ 93us) server_negotiation.cc:299] Negotiation successful
0618 23:30:05.475367 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":92,"threads_started":1}
I20260618 23:30:05.476346 26542 catalog_manager.cc:1347] Generated new cluster ID: d1e19d3f54bb455d99a5d1e5d58052a0
I20260618 23:30:05.476401 26542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:30:05.491264 26542 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:30:05.491827 26542 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:30:05.508436 26542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b20db0448a674518a2bbc8147e2422f0: Generated new TSK 0
I20260618 23:30:05.508661 26542 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:30:05.516420 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.129:0
--local_ip_for_outbound_sockets=127.8.39.129
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:41697
--builtin_ntp_servers=127.8.39.148:39455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 23:30:05.630438 26554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:05.630649 26554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:05.630676 26554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:05.631930 26554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:05.631974 26554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:05.632004 26554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.129
I20260618 23:30:05.633519 26554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:39455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:41697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.129
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:05.633778 26554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:05.634083 26554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:05.639519 26562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:05.639683 26559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:05.639683 26560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:05.640475 26554 server_base.cc:1034] running on GCE node
I20260618 23:30:05.640694 26554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:05.640955 26554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:05.642131 26554 hybrid_clock.cc:648] HybridClock initialized: now 1781825405642115 us; error 33 us; skew 500 ppm
I20260618 23:30:05.643874 26554 webserver.cc:460] Webserver started at http://127.8.39.129:46677/ using document root <none> and password file <none>
I20260618 23:30:05.644392 26554 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:05.644531 26554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:05.644698 26554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:05.645933 26554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-0/data/instance:
uuid: "75e30c3907b442ac9f7ed87f1ffdb0cc"
format_stamp: "Formatted at 2026-06-18 23:30:05 on dist-test-slave-47qt"
server_key: "37b2a1810c0a892142fb7f0258151b3a"
server_key_iv: "4577f6a4a97efd1a44f8e005c8e140a2"
server_key_version: "encryptionkey@0"
I20260618 23:30:05.646307 26554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance:
uuid: "75e30c3907b442ac9f7ed87f1ffdb0cc"
format_stamp: "Formatted at 2026-06-18 23:30:05 on dist-test-slave-47qt"
server_key: "37b2a1810c0a892142fb7f0258151b3a"
server_key_iv: "4577f6a4a97efd1a44f8e005c8e140a2"
server_key_version: "encryptionkey@0"
I20260618 23:30:05.648172 26554 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260618 23:30:05.649096 26568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:05.649488 26554 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:30:05.649653 26554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-0/wal
uuid: "75e30c3907b442ac9f7ed87f1ffdb0cc"
format_stamp: "Formatted at 2026-06-18 23:30:05 on dist-test-slave-47qt"
server_key: "37b2a1810c0a892142fb7f0258151b3a"
server_key_iv: "4577f6a4a97efd1a44f8e005c8e140a2"
server_key_version: "encryptionkey@0"
I20260618 23:30:05.649770 26554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:05.670696 26554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:05.671072 26554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:05.671236 26554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:05.671496 26554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:30:05.671854 26554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:30:05.671906 26554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:05.671947 26554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:30:05.671984 26554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:05.678699 26554 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.129:41045
I20260618 23:30:05.678740 26681 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.129:41045 every 8 connection(s)
I20260618 23:30:05.679095 26554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
I20260618 23:30:05.683547 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 26554
I20260618 23:30:05.683679 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance
I20260618 23:30:05.683928 8350 external_mini_cluster.cc:1428] Setting key 1d988bab2620a30b68d15528723f3110
I20260618 23:30:05.684659 26534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:05.679878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:55795 (local address 127.8.39.190:41697)
0618 23:30:05.680008 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:05.680010 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:05.680186 (+ 176us) server_negotiation.cc:407] Connection header received
0618 23:30:05.680728 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:05.680729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:05.680747 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:05.680801 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:05.681564 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.682076 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.683058 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.683262 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.683372 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:05.684363 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:05.684364 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:05.684364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:05.684369 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:05.684389 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:05.684392 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:05.684455 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:05.684553 (+ 98us) server_negotiation.cc:299] Negotiation successful
0618 23:30:05.684579 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260618 23:30:05.684876 26684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:05.679802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41697 (local address 127.8.39.129:55795)
0618 23:30:05.680058 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:05.680104 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:05.680626 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:05.680879 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:05.680883 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:05.680900 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:05.681473 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:05.681478 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.682185 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.682186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:05.682967 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:05.682970 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.683379 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.683381 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:05.683511 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:05.684169 (+ 658us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:30:05.684169 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:05.684209 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:30:05.684242 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:05.684456 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:05.684462 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:05.684466 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:30:05.684554 (+ 88us) client_negotiation.cc:240] Negotiation successful
0618 23:30:05.684653 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":96,"threads_started":1}
I20260618 23:30:05.684927 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.130:0
--local_ip_for_outbound_sockets=127.8.39.130
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:41697
--builtin_ntp_servers=127.8.39.148:39455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260618 23:30:05.685561 26682 heartbeater.cc:347] Connected to a master server at 127.8.39.190:41697
I20260618 23:30:05.685727 26682 heartbeater.cc:464] Registering TS with master...
I20260618 23:30:05.685943 26682 heartbeater.cc:511] Master 127.8.39.190:41697 requested a full tablet report, sending...
I20260618 23:30:05.686532 26494 ts_manager.cc:194] Registered new tserver with Master: 75e30c3907b442ac9f7ed87f1ffdb0cc (127.8.39.129:41045)
I20260618 23:30:05.687151 26494 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.39.129:55795
W20260618 23:30:05.772143 26685 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:05.772396 26685 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:05.772421 26685 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:05.773742 26685 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:05.773795 26685 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:05.773825 26685 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.130
I20260618 23:30:05.775305 26685 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:39455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:41697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26685
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.130
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:05.775532 26685 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:05.775805 26685 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:05.781055 26690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:05.781270 26685 server_base.cc:1034] running on GCE node
W20260618 23:30:05.781284 26691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:05.781373 26693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:05.781700 26685 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:05.782048 26685 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:05.783236 26685 hybrid_clock.cc:648] HybridClock initialized: now 1781825405783224 us; error 28 us; skew 500 ppm
I20260618 23:30:05.784700 26685 webserver.cc:460] Webserver started at http://127.8.39.130:33297/ using document root <none> and password file <none>
I20260618 23:30:05.784924 26685 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:05.784965 26685 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:05.785058 26685 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:05.786196 26685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-1/data/instance:
uuid: "6c9b8570a4e441efb9c072ca47611f28"
format_stamp: "Formatted at 2026-06-18 23:30:05 on dist-test-slave-47qt"
server_key: "89c7226462f8e90a7a9437cd3411149c"
server_key_iv: "a99423b55afdc2c8c343947441714375"
server_key_version: "encryptionkey@0"
I20260618 23:30:05.786495 26685 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance:
uuid: "6c9b8570a4e441efb9c072ca47611f28"
format_stamp: "Formatted at 2026-06-18 23:30:05 on dist-test-slave-47qt"
server_key: "89c7226462f8e90a7a9437cd3411149c"
server_key_iv: "a99423b55afdc2c8c343947441714375"
server_key_version: "encryptionkey@0"
I20260618 23:30:05.787886 26685 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260618 23:30:05.788727 26699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:05.788947 26685 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:30:05.789040 26685 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-1/wal
uuid: "6c9b8570a4e441efb9c072ca47611f28"
format_stamp: "Formatted at 2026-06-18 23:30:05 on dist-test-slave-47qt"
server_key: "89c7226462f8e90a7a9437cd3411149c"
server_key_iv: "a99423b55afdc2c8c343947441714375"
server_key_version: "encryptionkey@0"
I20260618 23:30:05.789101 26685 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:05.819579 26685 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:05.819975 26685 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:05.820343 26685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:05.820765 26685 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:30:05.821274 26685 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:30:05.821318 26685 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:05.821346 26685 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:30:05.821359 26685 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:05.827387 26685 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.130:45331
I20260618 23:30:05.827471 26812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.130:45331 every 8 connection(s)
I20260618 23:30:05.827823 26685 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
I20260618 23:30:05.831301 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 26685
I20260618 23:30:05.831435 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance
I20260618 23:30:05.831602 8350 external_mini_cluster.cc:1428] Setting key a3ed084e48d2c32050be1de71e3b3eb6
I20260618 23:30:05.832943 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.131:0
--local_ip_for_outbound_sockets=127.8.39.131
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:41697
--builtin_ntp_servers=127.8.39.148:39455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260618 23:30:05.834228 26534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:05.828760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:56525 (local address 127.8.39.190:41697)
0618 23:30:05.828855 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:05.828858 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:05.829448 (+ 590us) server_negotiation.cc:407] Connection header received
0618 23:30:05.829994 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:05.829996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:05.830006 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:05.830072 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:05.831012 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.831605 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.832692 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.832918 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.833003 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:05.833982 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:05.833984 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:05.833985 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:05.833990 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:05.834004 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:05.834008 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:05.834057 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:05.834124 (+ 67us) server_negotiation.cc:299] Negotiation successful
0618 23:30:05.834151 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260618 23:30:05.834461 26815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:05.828899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41697 (local address 127.8.39.130:56525)
0618 23:30:05.829276 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:05.829325 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:05.829877 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:05.830171 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:05.830175 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:05.830183 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:05.830842 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:05.830855 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.831730 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.831731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:05.832574 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:05.832576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.833012 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.833013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:05.833210 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:05.833834 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:30:05.833834 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:05.833864 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:30:05.833887 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:05.834056 (+ 169us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:05.834061 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:05.834063 (+ 2us) client_negotiation.cc:769] Sending connection context
0618 23:30:05.834130 (+ 67us) client_negotiation.cc:240] Negotiation successful
0618 23:30:05.834232 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"thread_start_us":141,"threads_started":1}
I20260618 23:30:05.834893 26813 heartbeater.cc:347] Connected to a master server at 127.8.39.190:41697
I20260618 23:30:05.834991 26813 heartbeater.cc:464] Registering TS with master...
I20260618 23:30:05.835194 26813 heartbeater.cc:511] Master 127.8.39.190:41697 requested a full tablet report, sending...
I20260618 23:30:05.835570 26494 ts_manager.cc:194] Registered new tserver with Master: 6c9b8570a4e441efb9c072ca47611f28 (127.8.39.130:45331)
I20260618 23:30:05.835966 26494 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.39.130:56525
W20260618 23:30:05.919549 26816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:05.919795 26816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:05.919821 26816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:05.921272 26816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:05.921327 26816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:05.921356 26816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.131
I20260618 23:30:05.922941 26816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:39455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:41697
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.131
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:05.923219 26816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:05.923550 26816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:05.928683 26821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:05.928812 26822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:05.928960 26824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:05.929311 26816 server_base.cc:1034] running on GCE node
I20260618 23:30:05.929510 26816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:05.929780 26816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:05.930955 26816 hybrid_clock.cc:648] HybridClock initialized: now 1781825405930941 us; error 28 us; skew 500 ppm
I20260618 23:30:05.932561 26816 webserver.cc:460] Webserver started at http://127.8.39.131:43257/ using document root <none> and password file <none>
I20260618 23:30:05.932809 26816 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:05.932879 26816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:05.933033 26816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:05.934226 26816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-2/data/instance:
uuid: "6e2711b25c4645f4a5217d32c7197d13"
format_stamp: "Formatted at 2026-06-18 23:30:05 on dist-test-slave-47qt"
server_key: "8a792ceef78593ae2401846db50520e0"
server_key_iv: "75bc0329638f9b691ec7df6793fdc778"
server_key_version: "encryptionkey@0"
I20260618 23:30:05.934609 26816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance:
uuid: "6e2711b25c4645f4a5217d32c7197d13"
format_stamp: "Formatted at 2026-06-18 23:30:05 on dist-test-slave-47qt"
server_key: "8a792ceef78593ae2401846db50520e0"
server_key_iv: "75bc0329638f9b691ec7df6793fdc778"
server_key_version: "encryptionkey@0"
I20260618 23:30:05.936168 26816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:30:05.937562 26830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:05.937808 26816 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:30:05.937925 26816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-2/wal
uuid: "6e2711b25c4645f4a5217d32c7197d13"
format_stamp: "Formatted at 2026-06-18 23:30:05 on dist-test-slave-47qt"
server_key: "8a792ceef78593ae2401846db50520e0"
server_key_iv: "75bc0329638f9b691ec7df6793fdc778"
server_key_version: "encryptionkey@0"
I20260618 23:30:05.938009 26816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:05.951025 26816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:05.951403 26816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:05.951570 26816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:05.951833 26816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:30:05.952311 26816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:30:05.952369 26816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:05.952414 26816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:30:05.952445 26816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:05.959185 26816 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.131:41751
I20260618 23:30:05.959295 26943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.131:41751 every 8 connection(s)
I20260618 23:30:05.959812 26816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
I20260618 23:30:05.966845 26534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:05.960727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:37885 (local address 127.8.39.190:41697)
0618 23:30:05.960801 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:05.960804 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:05.961191 (+ 387us) server_negotiation.cc:407] Connection header received
0618 23:30:05.961742 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:05.961743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:05.961753 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:05.961810 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:05.962578 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.963109 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.964951 (+ 1842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.965230 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.965297 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:05.966280 (+ 983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:05.966281 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:05.966281 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:05.966287 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:05.966301 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:05.966305 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:05.966376 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:05.966723 (+ 347us) server_negotiation.cc:299] Negotiation successful
0618 23:30:05.966751 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260618 23:30:05.967299 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:05.960706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:41697 (local address 127.8.39.131:37885)
0618 23:30:05.960966 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:05.961002 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:05.961599 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:05.961943 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:05.961946 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:05.961955 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:05.962463 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:05.962467 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.963246 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.963247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:05.964723 (+ 1476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:05.964727 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.965387 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.965388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:05.965519 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:05.966117 (+ 598us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:30:05.966118 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:05.966148 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:30:05.966173 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:05.966453 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:05.966465 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:05.966480 (+ 15us) client_negotiation.cc:769] Sending connection context
0618 23:30:05.966716 (+ 236us) client_negotiation.cc:240] Negotiation successful
0618 23:30:05.966881 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":111,"threads_started":1}
I20260618 23:30:05.968146 26944 heartbeater.cc:347] Connected to a master server at 127.8.39.190:41697
I20260618 23:30:05.968248 26944 heartbeater.cc:464] Registering TS with master...
I20260618 23:30:05.968406 26944 heartbeater.cc:511] Master 127.8.39.190:41697 requested a full tablet report, sending...
I20260618 23:30:05.968989 26494 ts_manager.cc:194] Registered new tserver with Master: 6e2711b25c4645f4a5217d32c7197d13 (127.8.39.131:41751)
I20260618 23:30:05.969165 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 26816
I20260618 23:30:05.969261 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance
I20260618 23:30:05.969461 8350 external_mini_cluster.cc:1428] Setting key a05306c4ddafb9840e2bae479f2f0aca
I20260618 23:30:05.969576 26494 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.39.131:37885
I20260618 23:30:05.971052 8350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:30:05.977401 26534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:05.972790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52784 (local address 127.8.39.190:41697)
0618 23:30:05.972914 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:05.972917 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:05.972983 (+ 66us) server_negotiation.cc:407] Connection header received
0618 23:30:05.973062 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:05.973063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:05.973073 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:05.973196 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:05.973388 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.974028 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.975217 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.975713 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.975826 (+ 113us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:05.976821 (+ 995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:05.976823 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:05.976823 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:05.976832 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:05.976859 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:05.976865 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:05.976989 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:05.977288 (+ 299us) server_negotiation.cc:299] Negotiation successful
0618 23:30:05.977304 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260618 23:30:05.979703 26494 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52784:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:30:05.981333 26494 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:30:05.988605 26955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:05.984020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60440 (local address 127.8.39.131:41751)
0618 23:30:05.984202 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:05.984206 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:05.984508 (+ 302us) server_negotiation.cc:407] Connection header received
0618 23:30:05.984577 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:05.984578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:05.984592 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:05.984695 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:30:05.984952 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.985862 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.987498 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.988074 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.988114 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:05.988213 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:05.988448 (+ 235us) server_negotiation.cc:299] Negotiation successful
0618 23:30:05.988491 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":114,"threads_started":1}
I20260618 23:30:05.988602 26953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:05.983938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:41045 (local address 127.0.0.1:45238)
0618 23:30:05.984718 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:05.984726 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:05.984774 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:05.985093 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:05.985093 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:05.985095 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:30:05.985155 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:05.985156 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.986389 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.986390 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:05.987329 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:05.987331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.988229 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.988231 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:05.988313 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:05.988339 (+ 26us) client_negotiation.cc:769] Sending connection context
0618 23:30:05.988417 (+ 78us) client_negotiation.cc:240] Negotiation successful
0618 23:30:05.988492 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":771,"mutex_wait_us":61,"thread_start_us":66,"threads_started":1}
I20260618 23:30:05.988602 26952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:05.983932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:41751 (local address 127.0.0.1:60440)
0618 23:30:05.984400 (+ 468us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:05.984439 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:05.984507 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:05.984724 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:05.984728 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:05.984731 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:30:05.984831 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:05.984834 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.985950 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.985951 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:05.987389 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:05.987391 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.988275 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.988276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:05.988346 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:05.988349 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:30:05.988417 (+ 68us) client_negotiation.cc:240] Negotiation successful
0618 23:30:05.988492 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":44,"thread_start_us":84,"threads_started":1}
I20260618 23:30:05.988889 26957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:05.984194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45238 (local address 127.8.39.129:41045)
0618 23:30:05.984385 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:05.984389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:05.984775 (+ 386us) server_negotiation.cc:407] Connection header received
0618 23:30:05.984844 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:05.984846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:05.984861 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:05.984931 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:30:05.985266 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.986310 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.987411 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.988126 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.988205 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:05.988302 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:05.988734 (+ 432us) server_negotiation.cc:299] Negotiation successful
0618 23:30:05.988783 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":96,"threads_started":1}
I20260618 23:30:05.989421 26956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:05.984193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39790 (local address 127.8.39.130:45331)
0618 23:30:05.984385 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:05.984389 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:05.984656 (+ 267us) server_negotiation.cc:407] Connection header received
0618 23:30:05.984711 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:05.984713 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:05.984727 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:05.984813 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:30:05.985009 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.986157 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.987581 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.988271 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.988333 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:05.988426 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:05.988659 (+ 233us) server_negotiation.cc:299] Negotiation successful
0618 23:30:05.989339 (+ 680us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":93,"threads_started":1}
I20260618 23:30:05.989615 26616 tablet_service.cc:1467] Processing CreateTablet for tablet 863f2ee6e95441b9aa83ba6e3e88ad29 (DEFAULT_TABLE table=test-table [id=4b4f56355e2242e9a7add4b8a6c7325f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:30:05.989616 26878 tablet_service.cc:1467] Processing CreateTablet for tablet 863f2ee6e95441b9aa83ba6e3e88ad29 (DEFAULT_TABLE table=test-table [id=4b4f56355e2242e9a7add4b8a6c7325f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:30:05.989984 26878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 863f2ee6e95441b9aa83ba6e3e88ad29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:05.989984 26616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 863f2ee6e95441b9aa83ba6e3e88ad29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:05.989154 26954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:05.983937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:45331 (local address 127.0.0.1:39790)
0618 23:30:05.984590 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:05.984597 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:05.984632 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:05.984876 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:05.984877 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:05.984880 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:30:05.984931 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:05.984933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.986255 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.986256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:05.987497 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:05.987499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:05.988461 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:05.988462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:05.988520 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:05.988524 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:30:05.988576 (+ 52us) client_negotiation.cc:240] Negotiation successful
0618 23:30:05.988618 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"mutex_wait_us":5,"spinlock_wait_cycles":768,"thread_start_us":69,"threads_started":1}
I20260618 23:30:05.993592 26958 tablet_bootstrap.cc:492] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 75e30c3907b442ac9f7ed87f1ffdb0cc: Bootstrap starting.
I20260618 23:30:05.993949 26747 tablet_service.cc:1467] Processing CreateTablet for tablet 863f2ee6e95441b9aa83ba6e3e88ad29 (DEFAULT_TABLE table=test-table [id=4b4f56355e2242e9a7add4b8a6c7325f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:30:05.994251 26747 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 863f2ee6e95441b9aa83ba6e3e88ad29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:05.994264 26958 tablet_bootstrap.cc:654] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 75e30c3907b442ac9f7ed87f1ffdb0cc: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:05.994572 26958 log.cc:824] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 75e30c3907b442ac9f7ed87f1ffdb0cc: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:05.994894 26959 tablet_bootstrap.cc:492] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13: Bootstrap starting.
I20260618 23:30:05.995328 26958 tablet_bootstrap.cc:492] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 75e30c3907b442ac9f7ed87f1ffdb0cc: No bootstrap required, opened a new log
I20260618 23:30:05.995399 26958 ts_tablet_manager.cc:1397] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 75e30c3907b442ac9f7ed87f1ffdb0cc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:30:05.995474 26959 tablet_bootstrap.cc:654] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:05.995826 26959 log.cc:824] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:05.996496 26959 tablet_bootstrap.cc:492] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13: No bootstrap required, opened a new log
I20260618 23:30:05.996587 26959 ts_tablet_manager.cc:1397] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:30:05.998490 26959 raft_consensus.cc:348] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e30c3907b442ac9f7ed87f1ffdb0cc" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41045 } } peers { permanent_uuid: "6c9b8570a4e441efb9c072ca47611f28" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 45331 } } peers { permanent_uuid: "6e2711b25c4645f4a5217d32c7197d13" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 41751 } }
I20260618 23:30:05.998695 26959 raft_consensus.cc:374] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:05.998862 26959 raft_consensus.cc:729] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e2711b25c4645f4a5217d32c7197d13, State: Initialized, Role: FOLLOWER
I20260618 23:30:05.998692 26958 raft_consensus.cc:348] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 75e30c3907b442ac9f7ed87f1ffdb0cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e30c3907b442ac9f7ed87f1ffdb0cc" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41045 } } peers { permanent_uuid: "6c9b8570a4e441efb9c072ca47611f28" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 45331 } } peers { permanent_uuid: "6e2711b25c4645f4a5217d32c7197d13" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 41751 } }
I20260618 23:30:05.998972 26958 raft_consensus.cc:374] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 75e30c3907b442ac9f7ed87f1ffdb0cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:05.999027 26958 raft_consensus.cc:729] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 75e30c3907b442ac9f7ed87f1ffdb0cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75e30c3907b442ac9f7ed87f1ffdb0cc, State: Initialized, Role: FOLLOWER
I20260618 23:30:05.998994 26959 consensus_queue.cc:260] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e30c3907b442ac9f7ed87f1ffdb0cc" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41045 } } peers { permanent_uuid: "6c9b8570a4e441efb9c072ca47611f28" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 45331 } } peers { permanent_uuid: "6e2711b25c4645f4a5217d32c7197d13" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 41751 } }
I20260618 23:30:05.998679 26962 tablet_bootstrap.cc:492] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6c9b8570a4e441efb9c072ca47611f28: Bootstrap starting.
I20260618 23:30:05.999162 26958 consensus_queue.cc:260] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 75e30c3907b442ac9f7ed87f1ffdb0cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e30c3907b442ac9f7ed87f1ffdb0cc" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41045 } } peers { permanent_uuid: "6c9b8570a4e441efb9c072ca47611f28" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 45331 } } peers { permanent_uuid: "6e2711b25c4645f4a5217d32c7197d13" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 41751 } }
I20260618 23:30:05.999308 26944 heartbeater.cc:503] Master 127.8.39.190:41697 was elected leader, sending a full tablet report...
I20260618 23:30:05.999426 26958 ts_tablet_manager.cc:1428] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 75e30c3907b442ac9f7ed87f1ffdb0cc: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260618 23:30:05.999444 26682 heartbeater.cc:503] Master 127.8.39.190:41697 was elected leader, sending a full tablet report...
I20260618 23:30:05.999275 26959 ts_tablet_manager.cc:1428] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:30:05.999979 26962 tablet_bootstrap.cc:654] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6c9b8570a4e441efb9c072ca47611f28: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:06.000352 26962 log.cc:824] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6c9b8570a4e441efb9c072ca47611f28: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:06.001173 26962 tablet_bootstrap.cc:492] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6c9b8570a4e441efb9c072ca47611f28: No bootstrap required, opened a new log
I20260618 23:30:06.001245 26962 ts_tablet_manager.cc:1397] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6c9b8570a4e441efb9c072ca47611f28: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260618 23:30:06.002983 26962 raft_consensus.cc:348] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6c9b8570a4e441efb9c072ca47611f28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e30c3907b442ac9f7ed87f1ffdb0cc" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41045 } } peers { permanent_uuid: "6c9b8570a4e441efb9c072ca47611f28" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 45331 } } peers { permanent_uuid: "6e2711b25c4645f4a5217d32c7197d13" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 41751 } }
I20260618 23:30:06.003114 26962 raft_consensus.cc:374] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6c9b8570a4e441efb9c072ca47611f28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:06.003135 26962 raft_consensus.cc:729] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6c9b8570a4e441efb9c072ca47611f28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c9b8570a4e441efb9c072ca47611f28, State: Initialized, Role: FOLLOWER
I20260618 23:30:06.003247 26962 consensus_queue.cc:260] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6c9b8570a4e441efb9c072ca47611f28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e30c3907b442ac9f7ed87f1ffdb0cc" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41045 } } peers { permanent_uuid: "6c9b8570a4e441efb9c072ca47611f28" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 45331 } } peers { permanent_uuid: "6e2711b25c4645f4a5217d32c7197d13" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 41751 } }
I20260618 23:30:06.003511 26962 ts_tablet_manager.cc:1428] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6c9b8570a4e441efb9c072ca47611f28: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260618 23:30:06.003711 26813 heartbeater.cc:503] Master 127.8.39.190:41697 was elected leader, sending a full tablet report...
W20260618 23:30:06.078415 26814 tablet.cc:2369] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6c9b8570a4e441efb9c072ca47611f28: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:30:06.160791 26963 raft_consensus.cc:482] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:30:06.160983 26963 raft_consensus.cc:504] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e30c3907b442ac9f7ed87f1ffdb0cc" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41045 } } peers { permanent_uuid: "6c9b8570a4e441efb9c072ca47611f28" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 45331 } } peers { permanent_uuid: "6e2711b25c4645f4a5217d32c7197d13" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 41751 } }
I20260618 23:30:06.161423 26963 leader_election.cc:288] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75e30c3907b442ac9f7ed87f1ffdb0cc (127.8.39.129:41045), 6c9b8570a4e441efb9c072ca47611f28 (127.8.39.130:45331)
I20260618 23:30:06.164860 26957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:06.161846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:35591 (local address 127.8.39.129:41045)
0618 23:30:06.161895 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:06.161897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:06.162028 (+ 131us) server_negotiation.cc:407] Connection header received
0618 23:30:06.162124 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:06.162126 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:06.162140 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:06.162200 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:30:06.162402 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:06.162957 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:06.164083 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:06.164497 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:06.164554 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:06.164561 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:06.164737 (+ 176us) server_negotiation.cc:299] Negotiation successful
0618 23:30:06.164775 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260618 23:30:06.164887 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:06.161697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:41045 (local address 127.8.39.131:35591)
0618 23:30:06.161876 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:06.161901 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:06.162026 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:06.162247 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:06.162249 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:06.162251 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:30:06.162328 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:06.162330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:06.163058 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:06.163059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:06.163980 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:06.163983 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:06.164591 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:06.164592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:06.164663 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:06.164668 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:30:06.164724 (+ 56us) client_negotiation.cc:240] Negotiation successful
0618 23:30:06.164800 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":51}
I20260618 23:30:06.165014 26956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:06.161853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:42147 (local address 127.8.39.130:45331)
0618 23:30:06.161895 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:06.161897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:06.162219 (+ 322us) server_negotiation.cc:407] Connection header received
0618 23:30:06.162250 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:06.162251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:06.162265 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:06.162363 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:30:06.162572 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:06.163136 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:06.164203 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:06.164627 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:06.164714 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:06.164722 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:06.164897 (+ 175us) server_negotiation.cc:299] Negotiation successful
0618 23:30:06.164940 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260618 23:30:06.165199 26967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:06.161697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:45331 (local address 127.8.39.131:42147)
0618 23:30:06.162098 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:06.162111 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:06.162187 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:06.162386 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:06.162387 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:06.162389 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:30:06.162467 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:06.162469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:06.163276 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:06.163277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:06.164102 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:06.164104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:06.164741 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:06.164742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:06.164811 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:06.164815 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:30:06.164873 (+ 58us) client_negotiation.cc:240] Negotiation successful
0618 23:30:06.165053 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":364,"mutex_wait_us":14,"thread_start_us":280,"threads_started":1}
I20260618 23:30:06.165274 26636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "863f2ee6e95441b9aa83ba6e3e88ad29" candidate_uuid: "6e2711b25c4645f4a5217d32c7197d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75e30c3907b442ac9f7ed87f1ffdb0cc" is_pre_election: true
I20260618 23:30:06.165459 26636 raft_consensus.cc:2454] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 75e30c3907b442ac9f7ed87f1ffdb0cc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e2711b25c4645f4a5217d32c7197d13 in term 0.
I20260618 23:30:06.165552 26767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "863f2ee6e95441b9aa83ba6e3e88ad29" candidate_uuid: "6e2711b25c4645f4a5217d32c7197d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c9b8570a4e441efb9c072ca47611f28" is_pre_election: true
I20260618 23:30:06.165702 26767 raft_consensus.cc:2454] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6c9b8570a4e441efb9c072ca47611f28 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e2711b25c4645f4a5217d32c7197d13 in term 0.
I20260618 23:30:06.165715 26832 leader_election.cc:302] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6e2711b25c4645f4a5217d32c7197d13, 75e30c3907b442ac9f7ed87f1ffdb0cc; no voters:
I20260618 23:30:06.165855 26963 raft_consensus.cc:2781] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:30:06.165917 26963 raft_consensus.cc:482] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:30:06.165942 26963 raft_consensus.cc:3037] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:06.166641 26963 raft_consensus.cc:504] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e30c3907b442ac9f7ed87f1ffdb0cc" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41045 } } peers { permanent_uuid: "6c9b8570a4e441efb9c072ca47611f28" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 45331 } } peers { permanent_uuid: "6e2711b25c4645f4a5217d32c7197d13" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 41751 } }
I20260618 23:30:06.166812 26963 leader_election.cc:288] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13 [CANDIDATE]: Term 1 election: Requested vote from peers 75e30c3907b442ac9f7ed87f1ffdb0cc (127.8.39.129:41045), 6c9b8570a4e441efb9c072ca47611f28 (127.8.39.130:45331)
I20260618 23:30:06.166946 26636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "863f2ee6e95441b9aa83ba6e3e88ad29" candidate_uuid: "6e2711b25c4645f4a5217d32c7197d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75e30c3907b442ac9f7ed87f1ffdb0cc"
I20260618 23:30:06.166972 26767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "863f2ee6e95441b9aa83ba6e3e88ad29" candidate_uuid: "6e2711b25c4645f4a5217d32c7197d13" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c9b8570a4e441efb9c072ca47611f28"
I20260618 23:30:06.167030 26767 raft_consensus.cc:3037] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6c9b8570a4e441efb9c072ca47611f28 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:06.167024 26636 raft_consensus.cc:3037] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 75e30c3907b442ac9f7ed87f1ffdb0cc [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:06.167904 26636 raft_consensus.cc:2454] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 75e30c3907b442ac9f7ed87f1ffdb0cc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e2711b25c4645f4a5217d32c7197d13 in term 1.
I20260618 23:30:06.167904 26767 raft_consensus.cc:2454] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6c9b8570a4e441efb9c072ca47611f28 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e2711b25c4645f4a5217d32c7197d13 in term 1.
I20260618 23:30:06.168097 26834 leader_election.cc:302] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c9b8570a4e441efb9c072ca47611f28, 6e2711b25c4645f4a5217d32c7197d13; no voters:
I20260618 23:30:06.168223 26963 raft_consensus.cc:2781] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:30:06.168442 26963 raft_consensus.cc:686] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13 [term 1 LEADER]: Becoming Leader. State: Replica: 6e2711b25c4645f4a5217d32c7197d13, State: Running, Role: LEADER
I20260618 23:30:06.168540 26963 consensus_queue.cc:237] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e30c3907b442ac9f7ed87f1ffdb0cc" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41045 } } peers { permanent_uuid: "6c9b8570a4e441efb9c072ca47611f28" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 45331 } } peers { permanent_uuid: "6e2711b25c4645f4a5217d32c7197d13" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 41751 } }
I20260618 23:30:06.169394 26493 catalog_manager.cc:5495] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e2711b25c4645f4a5217d32c7197d13 (127.8.39.131). New cstate: current_term: 1 leader_uuid: "6e2711b25c4645f4a5217d32c7197d13" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e30c3907b442ac9f7ed87f1ffdb0cc" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41045 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6c9b8570a4e441efb9c072ca47611f28" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 45331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e2711b25c4645f4a5217d32c7197d13" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 41751 } health_report { overall_health: HEALTHY } } }
W20260618 23:30:06.179843 26683 tablet.cc:2369] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 75e30c3907b442ac9f7ed87f1ffdb0cc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 23:30:06.210480 26945 tablet.cc:2369] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:30:06.218075 26955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:06.214932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60454 (local address 127.8.39.131:41751)
0618 23:30:06.215031 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:06.215033 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:06.215042 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:30:06.215077 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:06.215078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:06.215097 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:06.215169 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:30:06.215293 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:06.215772 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:06.216655 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:06.216890 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:06.216959 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:06.217570 (+ 611us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:30:06.217701 (+ 131us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:30:06.217824 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:06.217986 (+ 162us) server_negotiation.cc:299] Negotiation successful
0618 23:30:06.217995 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260618 23:30:06.219817 26767 raft_consensus.cc:1261] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6c9b8570a4e441efb9c072ca47611f28 [term 1 FOLLOWER]: Refusing update from remote peer 6e2711b25c4645f4a5217d32c7197d13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:30:06.219817 26636 raft_consensus.cc:1261] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 75e30c3907b442ac9f7ed87f1ffdb0cc [term 1 FOLLOWER]: Refusing update from remote peer 6e2711b25c4645f4a5217d32c7197d13: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:30:06.220218 26963 consensus_queue.cc:1035] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "75e30c3907b442ac9f7ed87f1ffdb0cc" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 41045 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:30:06.220377 26968 consensus_queue.cc:1035] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c9b8570a4e441efb9c072ca47611f28" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 45331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:30:06.222685 26971 mvcc.cc:205] Tried to move back new op lower bound from 7298356863873536000 to 7298356863667048448. Current Snapshot: MvccSnapshot[applied={T|T < 7298356863873536000}]
I20260618 23:30:06.222978 26973 mvcc.cc:205] Tried to move back new op lower bound from 7298356863873536000 to 7298356863667048448. Current Snapshot: MvccSnapshot[applied={T|T < 7298356863873536000}]
I20260618 23:30:06.223883 26972 mvcc.cc:205] Tried to move back new op lower bound from 7298356863873536000 to 7298356863667048448. Current Snapshot: MvccSnapshot[applied={T|T < 7298356863873536000}]
I20260618 23:30:06.225317 26494 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:30:06.225400 26494 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52784:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:30:06.226119 26494 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b20db0448a674518a2bbc8147e2422f0: Sending DeleteTablet for 3 replicas of tablet 863f2ee6e95441b9aa83ba6e3e88ad29
I20260618 23:30:06.226436 26616 tablet_service.cc:1514] Processing DeleteTablet for tablet 863f2ee6e95441b9aa83ba6e3e88ad29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:30:06 UTC) from {username='slave'} at 127.0.0.1:45238
I20260618 23:30:06.226457 26747 tablet_service.cc:1514] Processing DeleteTablet for tablet 863f2ee6e95441b9aa83ba6e3e88ad29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:30:06 UTC) from {username='slave'} at 127.0.0.1:39790
I20260618 23:30:06.226657 26878 tablet_service.cc:1514] Processing DeleteTablet for tablet 863f2ee6e95441b9aa83ba6e3e88ad29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:30:06 UTC) from {username='slave'} at 127.0.0.1:60440
I20260618 23:30:06.226871 26982 tablet_replica.cc:330] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13: stopping tablet replica
I20260618 23:30:06.226966 26982 raft_consensus.cc:2229] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13 [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:30:06.227088 26982 raft_consensus.cc:2258] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:30:06.227147 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 26554
I20260618 23:30:06.227491 26982 ts_tablet_manager.cc:1905] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:30:06.228997 26982 ts_tablet_manager.cc:1918] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:30:06.229108 26982 log.cc:1196] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-2/wal/wals/863f2ee6e95441b9aa83ba6e3e88ad29
I20260618 23:30:06.229404 26982 ts_tablet_manager.cc:1939] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6e2711b25c4645f4a5217d32c7197d13: Deleting consensus metadata
I20260618 23:30:06.229928 26478 catalog_manager.cc:4841] TS 6e2711b25c4645f4a5217d32c7197d13 (127.8.39.131:41751): tablet 863f2ee6e95441b9aa83ba6e3e88ad29 (table test-table [id=4b4f56355e2242e9a7add4b8a6c7325f]) successfully deleted
I20260618 23:30:06.233296 26981 tablet_replica.cc:330] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6c9b8570a4e441efb9c072ca47611f28: stopping tablet replica
I20260618 23:30:06.233462 26981 raft_consensus.cc:2229] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6c9b8570a4e441efb9c072ca47611f28 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:30:06.233603 26981 raft_consensus.cc:2258] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6c9b8570a4e441efb9c072ca47611f28 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:30:06.234246 26981 ts_tablet_manager.cc:1905] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6c9b8570a4e441efb9c072ca47611f28: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:30:06.236135 26981 ts_tablet_manager.cc:1918] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6c9b8570a4e441efb9c072ca47611f28: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:30:06.236217 26981 log.cc:1196] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6c9b8570a4e441efb9c072ca47611f28: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781825053542692-8350-0/minicluster-data/ts-1/wal/wals/863f2ee6e95441b9aa83ba6e3e88ad29
I20260618 23:30:06.236495 26981 ts_tablet_manager.cc:1939] T 863f2ee6e95441b9aa83ba6e3e88ad29 P 6c9b8570a4e441efb9c072ca47611f28: Deleting consensus metadata
W20260618 23:30:06.237097 26479 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.39.129:41045 (error 108)
I20260618 23:30:06.237272 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 26685
I20260618 23:30:06.238039 26952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:06.237924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:41045 (local address 127.0.0.1:45240)
0618 23:30:06.237967 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:06.238010 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.129:41045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260618 23:30:06.238247 26479 catalog_manager.cc:4584] TS 75e30c3907b442ac9f7ed87f1ffdb0cc (127.8.39.129:41045): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 863f2ee6e95441b9aa83ba6e3e88ad29: Network error: Client connection negotiation failed: client connection to 127.8.39.129:41045: connect: Connection refused (error 111)
I20260618 23:30:06.244138 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 26816
I20260618 23:30:06.244274 26952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:06.244117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:45331 (local address 127.0.0.1:39792)
0618 23:30:06.244178 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:06.244231 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.130:45331: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260618 23:30:06.249738 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 26463
2026-06-18T23:30:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (944 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:30:06 dist-test-slave-47qt krb5kdc[26986](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:30:06 dist-test-slave-47qt krb5kdc[26986](info): set up 2 sockets
Jun 18 23:30:06 dist-test-slave-47qt krb5kdc[26986](info): commencing operation
krb5kdc: starting...
W20260618 23:30:07.321830 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:30:07 dist-test-slave-47qt krb5kdc[26986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:30:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:30:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.39.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
I20260618 23:30:07.452000 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:36467
--webserver_interface=127.8.39.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:36467
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:30:07.539842 26999 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:07.540052 26999 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:07.540076 26999 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:07.541267 26999 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:30:07.541311 26999 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:07.541326 26999 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:07.541339 26999 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:30:07.541352 26999 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:30:07.542866 26999 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:36467
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:36467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26999
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:07.543115 26999 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:07.543351 26999 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:07.548369 27005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:07.548381 27007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:07.548439 27004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:07.548679 26999 server_base.cc:1034] running on GCE node
I20260618 23:30:07.549111 26999 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:07.549638 26999 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:07.550832 26999 hybrid_clock.cc:648] HybridClock initialized: now 1781825407550772 us; error 77 us; skew 500 ppm
Jun 18 23:30:07 dist-test-slave-47qt krb5kdc[26986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825407, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:07.554486 26999 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:30:07.554881 26999 webserver.cc:460] Webserver started at http://127.8.39.190:43191/ using document root <none> and password file <none>
I20260618 23:30:07.555136 26999 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:07.555182 26999 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:07.555332 26999 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:07.556403 26999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "c2bc3eff370b481995f62fee3bf5c97c"
format_stamp: "Formatted at 2026-06-18 23:30:07 on dist-test-slave-47qt"
server_key: "b057b7035e2619b333e8784d1266e668"
server_key_iv: "eb1a6757ae3caf5f990423e52e911014"
server_key_version: "encryptionkey@0"
I20260618 23:30:07.556780 26999 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "c2bc3eff370b481995f62fee3bf5c97c"
format_stamp: "Formatted at 2026-06-18 23:30:07 on dist-test-slave-47qt"
server_key: "b057b7035e2619b333e8784d1266e668"
server_key_iv: "eb1a6757ae3caf5f990423e52e911014"
server_key_version: "encryptionkey@0"
I20260618 23:30:07.558287 26999 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:30:07.559108 27014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:07.559365 26999 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:30:07.559470 26999 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "c2bc3eff370b481995f62fee3bf5c97c"
format_stamp: "Formatted at 2026-06-18 23:30:07 on dist-test-slave-47qt"
server_key: "b057b7035e2619b333e8784d1266e668"
server_key_iv: "eb1a6757ae3caf5f990423e52e911014"
server_key_version: "encryptionkey@0"
I20260618 23:30:07.559551 26999 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:07.581187 26999 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:07.584780 26999 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:07.584933 26999 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:07.589196 26999 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:36467
I20260618 23:30:07.589228 27066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:36467 every 8 connection(s)
I20260618 23:30:07.589591 26999 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:30:07.590466 27067 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:07.592676 27067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2bc3eff370b481995f62fee3bf5c97c: Bootstrap starting.
I20260618 23:30:07.593322 27067 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c2bc3eff370b481995f62fee3bf5c97c: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:07.593564 27067 log.cc:824] T 00000000000000000000000000000000 P c2bc3eff370b481995f62fee3bf5c97c: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:07.594308 27067 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c2bc3eff370b481995f62fee3bf5c97c: No bootstrap required, opened a new log
I20260618 23:30:07.595865 27067 raft_consensus.cc:348] T 00000000000000000000000000000000 P c2bc3eff370b481995f62fee3bf5c97c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2bc3eff370b481995f62fee3bf5c97c" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 36467 } }
I20260618 23:30:07.596016 27067 raft_consensus.cc:374] T 00000000000000000000000000000000 P c2bc3eff370b481995f62fee3bf5c97c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:07.596040 27067 raft_consensus.cc:729] T 00000000000000000000000000000000 P c2bc3eff370b481995f62fee3bf5c97c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2bc3eff370b481995f62fee3bf5c97c, State: Initialized, Role: FOLLOWER
I20260618 23:30:07.596196 27067 consensus_queue.cc:260] T 00000000000000000000000000000000 P c2bc3eff370b481995f62fee3bf5c97c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2bc3eff370b481995f62fee3bf5c97c" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 36467 } }
I20260618 23:30:07.596283 27067 raft_consensus.cc:388] T 00000000000000000000000000000000 P c2bc3eff370b481995f62fee3bf5c97c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:30:07.596307 27067 raft_consensus.cc:482] T 00000000000000000000000000000000 P c2bc3eff370b481995f62fee3bf5c97c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:30:07.596338 27067 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c2bc3eff370b481995f62fee3bf5c97c [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:07.596915 27067 raft_consensus.cc:504] T 00000000000000000000000000000000 P c2bc3eff370b481995f62fee3bf5c97c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2bc3eff370b481995f62fee3bf5c97c" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 36467 } }
I20260618 23:30:07.597010 27067 leader_election.cc:302] T 00000000000000000000000000000000 P c2bc3eff370b481995f62fee3bf5c97c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c2bc3eff370b481995f62fee3bf5c97c; no voters:
I20260618 23:30:07.597208 27067 leader_election.cc:288] T 00000000000000000000000000000000 P c2bc3eff370b481995f62fee3bf5c97c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:30:07.597321 27070 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c2bc3eff370b481995f62fee3bf5c97c [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:30:07.597512 27070 raft_consensus.cc:686] T 00000000000000000000000000000000 P c2bc3eff370b481995f62fee3bf5c97c [term 1 LEADER]: Becoming Leader. State: Replica: c2bc3eff370b481995f62fee3bf5c97c, State: Running, Role: LEADER
I20260618 23:30:07.597617 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 26999
I20260618 23:30:07.597657 27070 consensus_queue.cc:237] T 00000000000000000000000000000000 P c2bc3eff370b481995f62fee3bf5c97c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2bc3eff370b481995f62fee3bf5c97c" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 36467 } }
I20260618 23:30:07.597729 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:30:07.597926 8350 external_mini_cluster.cc:1428] Setting key 9a7d9d29740c339919c25267384ccc42
I20260618 23:30:07.598080 27071 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2bc3eff370b481995f62fee3bf5c97c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c2bc3eff370b481995f62fee3bf5c97c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2bc3eff370b481995f62fee3bf5c97c" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 36467 } } }
I20260618 23:30:07.598258 27071 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2bc3eff370b481995f62fee3bf5c97c [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:07.598284 27072 sys_catalog.cc:455] T 00000000000000000000000000000000 P c2bc3eff370b481995f62fee3bf5c97c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c2bc3eff370b481995f62fee3bf5c97c. Latest consensus state: current_term: 1 leader_uuid: "c2bc3eff370b481995f62fee3bf5c97c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c2bc3eff370b481995f62fee3bf5c97c" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 36467 } } }
I20260618 23:30:07.598379 27072 sys_catalog.cc:458] T 00000000000000000000000000000000 P c2bc3eff370b481995f62fee3bf5c97c [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:07.598640 27067 sys_catalog.cc:564] T 00000000000000000000000000000000 P c2bc3eff370b481995f62fee3bf5c97c [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:30:07.598752 27074 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:30:07.599299 27074 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:30:07.606604 27074 catalog_manager.cc:1347] Generated new cluster ID: 21b114dfa59449758fb30f7134a236f5
I20260618 23:30:07.606988 27074 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 18 23:30:07 dist-test-slave-47qt krb5kdc[26986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:30:07.612495 27077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:07.598622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33634 (local address 127.8.39.190:36467)
0618 23:30:07.599054 (+ 432us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:07.599064 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:07.599099 (+ 35us) server_negotiation.cc:407] Connection header received
0618 23:30:07.599221 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:07.599228 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:07.599284 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:07.600201 (+ 917us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:07.600291 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:07.601656 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:07.603011 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:07.603937 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:07.604110 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:07.608299 (+ 4189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:07.608304 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:07.608304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:07.608331 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:07.610348 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:07.610795 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:07.610798 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:07.610799 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:07.610933 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:07.611131 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:07.611132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:07.611132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:07.611427 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:07.611572 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:07.611950 (+ 378us) server_negotiation.cc:299] Negotiation successful
0618 23:30:07.612246 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":81,"threads_started":1}
I20260618 23:30:07.612860 27074 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:30:07.613543 27074 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:30:07.622390 27074 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c2bc3eff370b481995f62fee3bf5c97c: Generated new TSK 0
I20260618 23:30:07.622592 27074 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.39.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
Entry for principal kudu/127.8.39.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab.
I20260618 23:30:07.656713 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.129:0
--local_ip_for_outbound_sockets=127.8.39.129
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:36467
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:30:07.742237 27093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:07.742451 27093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:07.742476 27093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:07.743700 27093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:07.743745 27093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:07.743774 27093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.129
I20260618 23:30:07.745486 27093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.129.keytab
--principal=kudu/127.8.39.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:36467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.129
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:07.745745 27093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:07.746026 27093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:07.751248 27098 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:07.751369 27099 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:07.751248 27101 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:07.751540 27093 server_base.cc:1034] running on GCE node
I20260618 23:30:07.751740 27093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:07.751959 27093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:07.753189 27093 hybrid_clock.cc:648] HybridClock initialized: now 1781825407753170 us; error 31 us; skew 500 ppm
Jun 18 23:30:07 dist-test-slave-47qt krb5kdc[26986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825407, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:07.755824 27093 init.cc:377] Logged in from keytab as kudu/127.8.39.129@KRBTEST.COM (short username kudu)
I20260618 23:30:07.756641 27093 webserver.cc:460] Webserver started at http://127.8.39.129:45563/ using document root <none> and password file <none>
I20260618 23:30:07.757076 27093 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:07.757311 27093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:07.757488 27093 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:07.758585 27093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-0/data/instance:
uuid: "d9e7d40e8f1c4ff59a823315d295495c"
format_stamp: "Formatted at 2026-06-18 23:30:07 on dist-test-slave-47qt"
server_key: "5351ccec2727940a04cec640b21c2b16"
server_key_iv: "ca038b2eda3be99d48c0f1eaced60f09"
server_key_version: "encryptionkey@0"
I20260618 23:30:07.758951 27093 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance:
uuid: "d9e7d40e8f1c4ff59a823315d295495c"
format_stamp: "Formatted at 2026-06-18 23:30:07 on dist-test-slave-47qt"
server_key: "5351ccec2727940a04cec640b21c2b16"
server_key_iv: "ca038b2eda3be99d48c0f1eaced60f09"
server_key_version: "encryptionkey@0"
I20260618 23:30:07.760330 27093 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:30:07.761245 27108 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:07.761431 27093 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:30:07.761531 27093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-0/wal
uuid: "d9e7d40e8f1c4ff59a823315d295495c"
format_stamp: "Formatted at 2026-06-18 23:30:07 on dist-test-slave-47qt"
server_key: "5351ccec2727940a04cec640b21c2b16"
server_key_iv: "ca038b2eda3be99d48c0f1eaced60f09"
server_key_version: "encryptionkey@0"
I20260618 23:30:07.761621 27093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:07.769476 27093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:07.771750 27093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:07.771907 27093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:07.772135 27093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:30:07.772435 27093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:30:07.772495 27093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:07.772534 27093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:30:07.772562 27093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:07.778733 27093 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.129:39055
I20260618 23:30:07.778777 27221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.129:39055 every 8 connection(s)
I20260618 23:30:07.779094 27093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
I20260618 23:30:07.783396 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 27093
I20260618 23:30:07.783504 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance
I20260618 23:30:07.783725 8350 external_mini_cluster.cc:1428] Setting key 797be6c60d0dbe202ee4ec6a9836013c
Jun 18 23:30:07 dist-test-slave-47qt krb5kdc[26986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825407, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.129@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:30:07.791010 27077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:07.779822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:50989 (local address 127.8.39.190:36467)
0618 23:30:07.779922 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:07.779925 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:07.780098 (+ 173us) server_negotiation.cc:407] Connection header received
0618 23:30:07.780631 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:07.780635 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:07.780659 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:07.780764 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:07.781789 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:07.782433 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:07.783735 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:07.784100 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:07.784180 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:07.787786 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:07.787789 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:07.787789 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:07.787797 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:07.789457 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:07.789957 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:07.789959 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:07.789959 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:07.790029 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:07.790301 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:07.790302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:07.790302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:07.790441 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:07.790533 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:07.790817 (+ 284us) server_negotiation.cc:299] Negotiation successful
0618 23:30:07.790909 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260618 23:30:07.791229 27224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:07.779762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:36467 (local address 127.8.39.129:50989)
0618 23:30:07.779938 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:07.779979 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:07.780453 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:07.780849 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:07.780852 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:07.781448 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:07.781665 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:07.781669 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:07.782573 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:07.782574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:07.783639 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:07.783643 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:07.784205 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:07.784206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:07.784309 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:07.785019 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:30:07.785029 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:07.787495 (+ 2466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:07.789571 (+ 2076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:07.789580 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:07.789583 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:07.789868 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:07.790108 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:07.790108 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:07.790109 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:07.790225 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:07.790523 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:07.790531 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:07.790682 (+ 151us) client_negotiation.cc:769] Sending connection context
0618 23:30:07.790795 (+ 113us) client_negotiation.cc:240] Negotiation successful
0618 23:30:07.790981 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":105,"threads_started":1}
I20260618 23:30:07.791822 27222 heartbeater.cc:347] Connected to a master server at 127.8.39.190:36467
I20260618 23:30:07.791953 27222 heartbeater.cc:464] Registering TS with master...
I20260618 23:30:07.792181 27222 heartbeater.cc:511] Master 127.8.39.190:36467 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.8.39.130@KRBTEST.COM; defaulting to no policy
I20260618 23:30:07.792907 27031 ts_manager.cc:194] Registered new tserver with Master: d9e7d40e8f1c4ff59a823315d295495c (127.8.39.129:39055)
I20260618 23:30:07.794219 27031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.129@KRBTEST.COM'} at 127.8.39.129:50989
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
Entry for principal kudu/127.8.39.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab.
I20260618 23:30:07.812598 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.130:0
--local_ip_for_outbound_sockets=127.8.39.130
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:36467
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:30:07.906713 27227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:07.906946 27227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:07.906975 27227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:07.908325 27227 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:07.908377 27227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:07.908408 27227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.130
I20260618 23:30:07.909924 27227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.130.keytab
--principal=kudu/127.8.39.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:36467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27227
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.130
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:07.910179 27227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:07.910439 27227 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:07.916167 27235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:07.916195 27232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:07.916195 27233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:07.916435 27227 server_base.cc:1034] running on GCE node
I20260618 23:30:07.916873 27227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:07.917188 27227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:07.918378 27227 hybrid_clock.cc:648] HybridClock initialized: now 1781825407918329 us; error 69 us; skew 500 ppm
Jun 18 23:30:07 dist-test-slave-47qt krb5kdc[26986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825407, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:07.921727 27227 init.cc:377] Logged in from keytab as kudu/127.8.39.130@KRBTEST.COM (short username kudu)
I20260618 23:30:07.922236 27227 webserver.cc:460] Webserver started at http://127.8.39.130:38169/ using document root <none> and password file <none>
I20260618 23:30:07.922488 27227 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:07.922562 27227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:07.922724 27227 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:07.924432 27227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-1/data/instance:
uuid: "492d689d7b0449af92f2d448e8d3051a"
format_stamp: "Formatted at 2026-06-18 23:30:07 on dist-test-slave-47qt"
server_key: "2e5aa3fa7043ed3b94a01264f5b7a61b"
server_key_iv: "5febbd70509f1b51f8816ec5eaa900d9"
server_key_version: "encryptionkey@0"
I20260618 23:30:07.924983 27227 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance:
uuid: "492d689d7b0449af92f2d448e8d3051a"
format_stamp: "Formatted at 2026-06-18 23:30:07 on dist-test-slave-47qt"
server_key: "2e5aa3fa7043ed3b94a01264f5b7a61b"
server_key_iv: "5febbd70509f1b51f8816ec5eaa900d9"
server_key_version: "encryptionkey@0"
I20260618 23:30:07.926785 27227 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.001s
I20260618 23:30:07.927892 27242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:07.928143 27227 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:30:07.928260 27227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-1/wal
uuid: "492d689d7b0449af92f2d448e8d3051a"
format_stamp: "Formatted at 2026-06-18 23:30:07 on dist-test-slave-47qt"
server_key: "2e5aa3fa7043ed3b94a01264f5b7a61b"
server_key_iv: "5febbd70509f1b51f8816ec5eaa900d9"
server_key_version: "encryptionkey@0"
I20260618 23:30:07.928357 27227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:07.945094 27227 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:07.947527 27227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:07.947710 27227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:07.947963 27227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:30:07.948520 27227 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:30:07.948606 27227 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:07.948691 27227 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:30:07.948716 27227 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:07.955253 27227 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.130:42005
I20260618 23:30:07.955334 27355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.130:42005 every 8 connection(s)
I20260618 23:30:07.955687 27227 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
I20260618 23:30:07.959116 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 27227
I20260618 23:30:07.959239 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance
I20260618 23:30:07.959429 8350 external_mini_cluster.cc:1428] Setting key 047089d05a69c711be8a384edf9d8c31
Jun 18 23:30:07 dist-test-slave-47qt krb5kdc[26986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825407, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.130@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.39.131@KRBTEST.COM; defaulting to no policy
I20260618 23:30:07.968065 27077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:07.956573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:35687 (local address 127.8.39.190:36467)
0618 23:30:07.956662 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:07.956665 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:07.956828 (+ 163us) server_negotiation.cc:407] Connection header received
0618 23:30:07.957481 (+ 653us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:07.957482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:07.957494 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:07.957743 (+ 249us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:07.958960 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:07.959709 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:07.960981 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:07.961330 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:07.961404 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:07.964632 (+ 3228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:07.964633 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:07.964634 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:07.964641 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:07.966170 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:07.966667 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:07.966668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:07.966669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:07.966728 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:07.967000 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:07.967001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:07.967002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:07.967120 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:07.967197 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:07.967875 (+ 678us) server_negotiation.cc:299] Negotiation successful
0618 23:30:07.967965 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260618 23:30:07.968539 27358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:07.956518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:36467 (local address 127.8.39.130:35687)
0618 23:30:07.956707 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:07.956743 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:07.957313 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:07.957824 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:07.957828 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:07.958610 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:07.958803 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:07.958808 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:07.959923 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:07.959924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:07.960893 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:07.960896 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:07.961621 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:07.961622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:07.961796 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:07.962618 (+ 822us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:30:07.962629 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:07.964539 (+ 1910us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:07.966284 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:07.966291 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:07.966294 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:07.966553 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:07.966801 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:07.966802 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:07.966802 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:07.966920 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:07.967188 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:07.967195 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:07.967499 (+ 304us) client_negotiation.cc:769] Sending connection context
0618 23:30:07.968188 (+ 689us) client_negotiation.cc:240] Negotiation successful
0618 23:30:07.968357 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":99,"threads_started":1}
I20260618 23:30:07.969013 27356 heartbeater.cc:347] Connected to a master server at 127.8.39.190:36467
I20260618 23:30:07.969106 27356 heartbeater.cc:464] Registering TS with master...
I20260618 23:30:07.969313 27356 heartbeater.cc:511] Master 127.8.39.190:36467 requested a full tablet report, sending...
I20260618 23:30:07.969884 27031 ts_manager.cc:194] Registered new tserver with Master: 492d689d7b0449af92f2d448e8d3051a (127.8.39.130:42005)
I20260618 23:30:07.970292 27031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.130@KRBTEST.COM'} at 127.8.39.130:35687
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
Entry for principal kudu/127.8.39.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab.
I20260618 23:30:07.987950 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.131:0
--local_ip_for_outbound_sockets=127.8.39.131
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:36467
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:32833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:30:08.074860 27361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:08.075080 27361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:08.075107 27361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:08.076217 27361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:08.076257 27361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:08.076283 27361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.131
I20260618 23:30:08.077795 27361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:32833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.131.keytab
--principal=kudu/127.8.39.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:36467
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27361
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.131
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:08.077993 27361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:08.078176 27361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:08.083294 27367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:08.083398 27366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:08.083410 27369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:08.083858 27361 server_base.cc:1034] running on GCE node
I20260618 23:30:08.084115 27361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:08.084378 27361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:08.085577 27361 hybrid_clock.cc:648] HybridClock initialized: now 1781825408085591 us; error 59 us; skew 500 ppm
Jun 18 23:30:08 dist-test-slave-47qt krb5kdc[26986](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825408, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:08.088359 27361 init.cc:377] Logged in from keytab as kudu/127.8.39.131@KRBTEST.COM (short username kudu)
I20260618 23:30:08.088862 27361 webserver.cc:460] Webserver started at http://127.8.39.131:34435/ using document root <none> and password file <none>
I20260618 23:30:08.089099 27361 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:08.089224 27361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:08.089378 27361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:08.090473 27361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-2/data/instance:
uuid: "3336603f41764295a511dbb65169c654"
format_stamp: "Formatted at 2026-06-18 23:30:08 on dist-test-slave-47qt"
server_key: "75299d88dad2485a617435f7b8f599d3"
server_key_iv: "ba995de58df46c8fec721886b135b4d2"
server_key_version: "encryptionkey@0"
I20260618 23:30:08.090865 27361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance:
uuid: "3336603f41764295a511dbb65169c654"
format_stamp: "Formatted at 2026-06-18 23:30:08 on dist-test-slave-47qt"
server_key: "75299d88dad2485a617435f7b8f599d3"
server_key_iv: "ba995de58df46c8fec721886b135b4d2"
server_key_version: "encryptionkey@0"
I20260618 23:30:08.092399 27361 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:30:08.093456 27376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:08.093828 27361 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:30:08.093971 27361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-2/wal
uuid: "3336603f41764295a511dbb65169c654"
format_stamp: "Formatted at 2026-06-18 23:30:08 on dist-test-slave-47qt"
server_key: "75299d88dad2485a617435f7b8f599d3"
server_key_iv: "ba995de58df46c8fec721886b135b4d2"
server_key_version: "encryptionkey@0"
I20260618 23:30:08.094091 27361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:08.115419 27361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:08.118079 27361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:08.118481 27361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:08.118871 27361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:30:08.119583 27361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:30:08.119692 27361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:08.119735 27361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:30:08.119768 27361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:08.127970 27361 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.131:41325
I20260618 23:30:08.128018 27489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.131:41325 every 8 connection(s)
I20260618 23:30:08.128449 27361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
I20260618 23:30:08.133361 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 27361
I20260618 23:30:08.133489 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance
I20260618 23:30:08.133700 8350 external_mini_cluster.cc:1428] Setting key 5f03b7a2f0f862704b5e1fdd92dfb3f9
Jun 18 23:30:08 dist-test-slave-47qt krb5kdc[26986](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825408, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.131@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:30:08.139083 27077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:08.129497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:34999 (local address 127.8.39.190:36467)
0618 23:30:08.129564 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:08.129596 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:08.129804 (+ 208us) server_negotiation.cc:407] Connection header received
0618 23:30:08.130387 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:08.130388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:08.130400 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:08.130456 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:08.131549 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.132056 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.133134 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.133354 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.133440 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:08.136037 (+ 2597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:08.136039 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:08.136040 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:08.136054 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:08.137614 (+ 1560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:08.138076 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:08.138077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:08.138078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:08.138137 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:08.138414 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:08.138415 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:08.138415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:08.138540 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:08.138617 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:08.138897 (+ 280us) server_negotiation.cc:299] Negotiation successful
0618 23:30:08.138978 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260618 23:30:08.139288 27492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:08.129417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:36467 (local address 127.8.39.131:34999)
0618 23:30:08.129672 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:08.129712 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:08.130275 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:08.130568 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:08.130572 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:08.131176 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:08.131435 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:08.131440 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.132153 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.132154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:08.133023 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:08.133026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.133459 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.133460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:08.133629 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:08.134235 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:30:08.134244 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:08.135879 (+ 1635us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:08.137724 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:08.137732 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:08.137735 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:08.137974 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:08.138209 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:08.138209 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:08.138210 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:08.138338 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:08.138608 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:08.138614 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:08.138763 (+ 149us) client_negotiation.cc:769] Sending connection context
0618 23:30:08.138875 (+ 112us) client_negotiation.cc:240] Negotiation successful
0618 23:30:08.139045 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":110,"threads_started":1}
I20260618 23:30:08.139813 27490 heartbeater.cc:347] Connected to a master server at 127.8.39.190:36467
I20260618 23:30:08.139897 27490 heartbeater.cc:464] Registering TS with master...
I20260618 23:30:08.140048 27490 heartbeater.cc:511] Master 127.8.39.190:36467 requested a full tablet report, sending...
I20260618 23:30:08.140398 27031 ts_manager.cc:194] Registered new tserver with Master: 3336603f41764295a511dbb65169c654 (127.8.39.131:41325)
I20260618 23:30:08.140791 27031 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.39.131@KRBTEST.COM'} at 127.8.39.131:34999
I20260618 23:30:08.145530 8350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:30:08.153275 27077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:08.146848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33642 (local address 127.8.39.190:36467)
0618 23:30:08.146961 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:08.146963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:08.147085 (+ 122us) server_negotiation.cc:407] Connection header received
0618 23:30:08.147176 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:08.147177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:08.147187 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:08.147261 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:08.147809 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.148286 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.149030 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.149292 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.149375 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:08.150600 (+ 1225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:08.150602 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:08.150602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:08.150608 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:08.152033 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:08.152398 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:08.152400 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:08.152400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:08.152462 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:08.152664 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:08.152665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:08.152665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:08.152818 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:08.152930 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:08.153042 (+ 112us) server_negotiation.cc:299] Negotiation successful
0618 23:30:08.153147 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260618 23:30:08.154891 27031 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33642:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:30:08.156447 27031 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:30:08.163414 27504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:08.159239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:41325 (local address 127.0.0.1:50514)
0618 23:30:08.159550 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:08.159559 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:08.159593 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:08.160067 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:08.160068 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:08.160070 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:30:08.160116 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:08.160118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.161026 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.161027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:08.162334 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:08.162336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.163104 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.163105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:08.163184 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:08.163198 (+ 14us) client_negotiation.cc:769] Sending connection context
0618 23:30:08.163270 (+ 72us) client_negotiation.cc:240] Negotiation successful
0618 23:30:08.163325 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":10,"thread_start_us":123,"threads_started":1}
I20260618 23:30:08.163586 27502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:08.159266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35188 (local address 127.8.39.130:42005)
0618 23:30:08.159600 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:08.159604 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:08.159615 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:30:08.159654 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:08.159656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:08.159670 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:08.159743 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:30:08.160041 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.161248 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.162491 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.163168 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.163243 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:08.163297 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:08.163382 (+ 85us) server_negotiation.cc:299] Negotiation successful
0618 23:30:08.163495 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":78,"threads_started":1}
I20260618 23:30:08.164008 27505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:08.159510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50514 (local address 127.8.39.131:41325)
0618 23:30:08.159853 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:08.159856 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:08.159863 (+ 7us) server_negotiation.cc:407] Connection header received
0618 23:30:08.159894 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:08.159896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:08.159906 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:08.159996 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:30:08.160195 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.160921 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.162439 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.163010 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.163094 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:08.163156 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:08.163911 (+ 755us) server_negotiation.cc:299] Negotiation successful
0618 23:30:08.163949 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":87,"threads_started":1}
I20260618 23:30:08.163503 27501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:08.159165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:42005 (local address 127.0.0.1:35188)
0618 23:30:08.159321 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:08.159359 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:08.159439 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:08.159787 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:08.159791 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:08.159794 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:30:08.159931 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:08.159933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.161363 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.161364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:08.162332 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:08.162335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.163248 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.163249 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:08.163304 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:08.163307 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:30:08.163365 (+ 58us) client_negotiation.cc:240] Negotiation successful
0618 23:30:08.163397 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":5,"thread_start_us":59,"threads_started":1}
I20260618 23:30:08.164188 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:08.159525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35434 (local address 127.8.39.129:39055)
0618 23:30:08.159703 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:08.159707 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:08.159755 (+ 48us) server_negotiation.cc:407] Connection header received
0618 23:30:08.159897 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:08.159900 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:08.159927 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:08.160049 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:30:08.160259 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.161391 (+ 1132us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.162673 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.163666 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.163738 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:08.163793 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:08.163965 (+ 172us) server_negotiation.cc:299] Negotiation successful
0618 23:30:08.164032 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":77,"threads_started":1}
I20260618 23:30:08.164057 27503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:08.159239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:39055 (local address 127.0.0.1:35434)
0618 23:30:08.159424 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:08.159434 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:08.159478 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:08.160131 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:08.160132 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:08.160134 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:30:08.160182 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:08.160183 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.161490 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.161491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:08.162592 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:08.162594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.163822 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.163822 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:08.163876 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:08.163879 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:30:08.163952 (+ 73us) client_negotiation.cc:240] Negotiation successful
0618 23:30:08.163991 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":2,"thread_start_us":122,"threads_started":1}
I20260618 23:30:08.164852 27424 tablet_service.cc:1467] Processing CreateTablet for tablet 0c2b1c7a082a4d97a2ca90e3ee22e8a7 (DEFAULT_TABLE table=test-table [id=847cb5c20ea44ca386ddc9de25d86172]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:30:08.164852 27290 tablet_service.cc:1467] Processing CreateTablet for tablet 0c2b1c7a082a4d97a2ca90e3ee22e8a7 (DEFAULT_TABLE table=test-table [id=847cb5c20ea44ca386ddc9de25d86172]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:30:08.165107 27156 tablet_service.cc:1467] Processing CreateTablet for tablet 0c2b1c7a082a4d97a2ca90e3ee22e8a7 (DEFAULT_TABLE table=test-table [id=847cb5c20ea44ca386ddc9de25d86172]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:30:08.165234 27290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c2b1c7a082a4d97a2ca90e3ee22e8a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:08.165234 27424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c2b1c7a082a4d97a2ca90e3ee22e8a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:08.165470 27156 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c2b1c7a082a4d97a2ca90e3ee22e8a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:08.168540 27507 tablet_bootstrap.cc:492] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P d9e7d40e8f1c4ff59a823315d295495c: Bootstrap starting.
I20260618 23:30:08.168839 27509 tablet_bootstrap.cc:492] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 3336603f41764295a511dbb65169c654: Bootstrap starting.
I20260618 23:30:08.169291 27508 tablet_bootstrap.cc:492] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 492d689d7b0449af92f2d448e8d3051a: Bootstrap starting.
I20260618 23:30:08.169518 27507 tablet_bootstrap.cc:654] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P d9e7d40e8f1c4ff59a823315d295495c: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:08.169572 27509 tablet_bootstrap.cc:654] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 3336603f41764295a511dbb65169c654: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:08.169945 27508 tablet_bootstrap.cc:654] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 492d689d7b0449af92f2d448e8d3051a: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:08.170279 27509 log.cc:824] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 3336603f41764295a511dbb65169c654: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:08.170379 27507 log.cc:824] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P d9e7d40e8f1c4ff59a823315d295495c: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:08.170557 27508 log.cc:824] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 492d689d7b0449af92f2d448e8d3051a: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:08.171386 27508 tablet_bootstrap.cc:492] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 492d689d7b0449af92f2d448e8d3051a: No bootstrap required, opened a new log
I20260618 23:30:08.171448 27508 ts_tablet_manager.cc:1397] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 492d689d7b0449af92f2d448e8d3051a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260618 23:30:08.171629 27507 tablet_bootstrap.cc:492] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P d9e7d40e8f1c4ff59a823315d295495c: No bootstrap required, opened a new log
I20260618 23:30:08.171715 27507 ts_tablet_manager.cc:1397] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P d9e7d40e8f1c4ff59a823315d295495c: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260618 23:30:08.172863 27508 raft_consensus.cc:348] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 492d689d7b0449af92f2d448e8d3051a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492d689d7b0449af92f2d448e8d3051a" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42005 } } peers { permanent_uuid: "3336603f41764295a511dbb65169c654" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 41325 } } peers { permanent_uuid: "d9e7d40e8f1c4ff59a823315d295495c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 39055 } }
I20260618 23:30:08.173004 27508 raft_consensus.cc:374] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 492d689d7b0449af92f2d448e8d3051a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:08.173027 27508 raft_consensus.cc:729] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 492d689d7b0449af92f2d448e8d3051a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 492d689d7b0449af92f2d448e8d3051a, State: Initialized, Role: FOLLOWER
I20260618 23:30:08.173177 27508 consensus_queue.cc:260] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 492d689d7b0449af92f2d448e8d3051a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492d689d7b0449af92f2d448e8d3051a" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42005 } } peers { permanent_uuid: "3336603f41764295a511dbb65169c654" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 41325 } } peers { permanent_uuid: "d9e7d40e8f1c4ff59a823315d295495c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 39055 } }
I20260618 23:30:08.173128 27507 raft_consensus.cc:348] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P d9e7d40e8f1c4ff59a823315d295495c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492d689d7b0449af92f2d448e8d3051a" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42005 } } peers { permanent_uuid: "3336603f41764295a511dbb65169c654" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 41325 } } peers { permanent_uuid: "d9e7d40e8f1c4ff59a823315d295495c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 39055 } }
I20260618 23:30:08.173408 27507 raft_consensus.cc:374] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P d9e7d40e8f1c4ff59a823315d295495c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:08.173485 27507 raft_consensus.cc:729] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P d9e7d40e8f1c4ff59a823315d295495c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9e7d40e8f1c4ff59a823315d295495c, State: Initialized, Role: FOLLOWER
I20260618 23:30:08.173475 27356 heartbeater.cc:503] Master 127.8.39.190:36467 was elected leader, sending a full tablet report...
I20260618 23:30:08.173720 27507 consensus_queue.cc:260] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P d9e7d40e8f1c4ff59a823315d295495c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492d689d7b0449af92f2d448e8d3051a" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42005 } } peers { permanent_uuid: "3336603f41764295a511dbb65169c654" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 41325 } } peers { permanent_uuid: "d9e7d40e8f1c4ff59a823315d295495c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 39055 } }
I20260618 23:30:08.173473 27508 ts_tablet_manager.cc:1428] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 492d689d7b0449af92f2d448e8d3051a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260618 23:30:08.173893 27509 tablet_bootstrap.cc:492] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 3336603f41764295a511dbb65169c654: No bootstrap required, opened a new log
I20260618 23:30:08.174026 27509 ts_tablet_manager.cc:1397] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 3336603f41764295a511dbb65169c654: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260618 23:30:08.174022 27507 ts_tablet_manager.cc:1428] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P d9e7d40e8f1c4ff59a823315d295495c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260618 23:30:08.174073 27222 heartbeater.cc:503] Master 127.8.39.190:36467 was elected leader, sending a full tablet report...
I20260618 23:30:08.175771 27509 raft_consensus.cc:348] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 3336603f41764295a511dbb65169c654 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492d689d7b0449af92f2d448e8d3051a" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42005 } } peers { permanent_uuid: "3336603f41764295a511dbb65169c654" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 41325 } } peers { permanent_uuid: "d9e7d40e8f1c4ff59a823315d295495c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 39055 } }
I20260618 23:30:08.175899 27509 raft_consensus.cc:374] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 3336603f41764295a511dbb65169c654 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:08.175920 27509 raft_consensus.cc:729] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 3336603f41764295a511dbb65169c654 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3336603f41764295a511dbb65169c654, State: Initialized, Role: FOLLOWER
I20260618 23:30:08.175995 27509 consensus_queue.cc:260] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 3336603f41764295a511dbb65169c654 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492d689d7b0449af92f2d448e8d3051a" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42005 } } peers { permanent_uuid: "3336603f41764295a511dbb65169c654" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 41325 } } peers { permanent_uuid: "d9e7d40e8f1c4ff59a823315d295495c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 39055 } }
I20260618 23:30:08.176232 27509 ts_tablet_manager.cc:1428] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 3336603f41764295a511dbb65169c654: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260618 23:30:08.176292 27490 heartbeater.cc:503] Master 127.8.39.190:36467 was elected leader, sending a full tablet report...
W20260618 23:30:08.206326 27357 tablet.cc:2369] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 492d689d7b0449af92f2d448e8d3051a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:30:08.265813 27513 raft_consensus.cc:482] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 492d689d7b0449af92f2d448e8d3051a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:30:08.265997 27513 raft_consensus.cc:504] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 492d689d7b0449af92f2d448e8d3051a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492d689d7b0449af92f2d448e8d3051a" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42005 } } peers { permanent_uuid: "3336603f41764295a511dbb65169c654" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 41325 } } peers { permanent_uuid: "d9e7d40e8f1c4ff59a823315d295495c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 39055 } }
I20260618 23:30:08.266350 27513 leader_election.cc:288] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 492d689d7b0449af92f2d448e8d3051a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3336603f41764295a511dbb65169c654 (127.8.39.131:41325), d9e7d40e8f1c4ff59a823315d295495c (127.8.39.129:39055)
I20260618 23:30:08.269315 27505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:08.266665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:34319 (local address 127.8.39.131:41325)
0618 23:30:08.266712 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:08.266715 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:08.266828 (+ 113us) server_negotiation.cc:407] Connection header received
0618 23:30:08.266875 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:08.266875 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:08.266886 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:08.266957 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:30:08.267123 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.267619 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.268576 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.268930 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.268989 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:08.268996 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:08.269196 (+ 200us) server_negotiation.cc:299] Negotiation successful
0618 23:30:08.269244 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260618 23:30:08.269331 27516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:08.266543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:41325 (local address 127.8.39.130:34319)
0618 23:30:08.266761 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:08.266770 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:08.266819 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:08.266985 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:08.266986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:08.266988 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:30:08.267056 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:08.267057 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.267706 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.267706 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:08.268453 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:08.268462 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.269009 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.269010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:08.269081 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:08.269086 (+ 5us) client_negotiation.cc:769] Sending connection context
0618 23:30:08.269187 (+ 101us) client_negotiation.cc:240] Negotiation successful
0618 23:30:08.269227 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204,"mutex_wait_us":24,"thread_start_us":105,"threads_started":1}
I20260618 23:30:08.269399 27506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:08.266642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:32799 (local address 127.8.39.129:39055)
0618 23:30:08.266688 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:08.266691 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:08.266699 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:30:08.266728 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:08.266729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:08.266742 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:08.266802 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:30:08.267004 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.267528 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.268529 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.268985 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.269054 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:08.269062 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:08.269285 (+ 223us) server_negotiation.cc:299] Negotiation successful
0618 23:30:08.269323 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260618 23:30:08.269639 27358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:08.266528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:39055 (local address 127.8.39.130:32799)
0618 23:30:08.266583 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:08.266596 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:08.266646 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:08.266839 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:08.266841 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:08.266843 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:30:08.266920 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:08.266922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.267606 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.267607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:08.268431 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:08.268433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.269088 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.269089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:08.269205 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:08.269209 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:30:08.269264 (+ 55us) client_negotiation.cc:240] Negotiation successful
0618 23:30:08.269305 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":1}
I20260618 23:30:08.269709 27444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c2b1c7a082a4d97a2ca90e3ee22e8a7" candidate_uuid: "492d689d7b0449af92f2d448e8d3051a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3336603f41764295a511dbb65169c654" is_pre_election: true
I20260618 23:30:08.269878 27444 raft_consensus.cc:2454] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 3336603f41764295a511dbb65169c654 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492d689d7b0449af92f2d448e8d3051a in term 0.
I20260618 23:30:08.269966 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c2b1c7a082a4d97a2ca90e3ee22e8a7" candidate_uuid: "492d689d7b0449af92f2d448e8d3051a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9e7d40e8f1c4ff59a823315d295495c" is_pre_election: true
I20260618 23:30:08.270109 27176 raft_consensus.cc:2454] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P d9e7d40e8f1c4ff59a823315d295495c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 492d689d7b0449af92f2d448e8d3051a in term 0.
I20260618 23:30:08.270140 27244 leader_election.cc:302] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 492d689d7b0449af92f2d448e8d3051a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3336603f41764295a511dbb65169c654, 492d689d7b0449af92f2d448e8d3051a; no voters:
I20260618 23:30:08.270254 27513 raft_consensus.cc:2781] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 492d689d7b0449af92f2d448e8d3051a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:30:08.270328 27513 raft_consensus.cc:482] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 492d689d7b0449af92f2d448e8d3051a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:30:08.270352 27513 raft_consensus.cc:3037] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 492d689d7b0449af92f2d448e8d3051a [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:08.271073 27513 raft_consensus.cc:504] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 492d689d7b0449af92f2d448e8d3051a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492d689d7b0449af92f2d448e8d3051a" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42005 } } peers { permanent_uuid: "3336603f41764295a511dbb65169c654" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 41325 } } peers { permanent_uuid: "d9e7d40e8f1c4ff59a823315d295495c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 39055 } }
I20260618 23:30:08.271222 27513 leader_election.cc:288] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 492d689d7b0449af92f2d448e8d3051a [CANDIDATE]: Term 1 election: Requested vote from peers 3336603f41764295a511dbb65169c654 (127.8.39.131:41325), d9e7d40e8f1c4ff59a823315d295495c (127.8.39.129:39055)
I20260618 23:30:08.271346 27444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c2b1c7a082a4d97a2ca90e3ee22e8a7" candidate_uuid: "492d689d7b0449af92f2d448e8d3051a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3336603f41764295a511dbb65169c654"
I20260618 23:30:08.271431 27444 raft_consensus.cc:3037] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 3336603f41764295a511dbb65169c654 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:08.271566 27176 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c2b1c7a082a4d97a2ca90e3ee22e8a7" candidate_uuid: "492d689d7b0449af92f2d448e8d3051a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9e7d40e8f1c4ff59a823315d295495c"
I20260618 23:30:08.271726 27176 raft_consensus.cc:3037] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P d9e7d40e8f1c4ff59a823315d295495c [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:08.272282 27444 raft_consensus.cc:2454] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 3336603f41764295a511dbb65169c654 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492d689d7b0449af92f2d448e8d3051a in term 1.
I20260618 23:30:08.272483 27244 leader_election.cc:302] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 492d689d7b0449af92f2d448e8d3051a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3336603f41764295a511dbb65169c654, 492d689d7b0449af92f2d448e8d3051a; no voters:
I20260618 23:30:08.272591 27513 raft_consensus.cc:2781] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 492d689d7b0449af92f2d448e8d3051a [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:30:08.272748 27513 raft_consensus.cc:686] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 492d689d7b0449af92f2d448e8d3051a [term 1 LEADER]: Becoming Leader. State: Replica: 492d689d7b0449af92f2d448e8d3051a, State: Running, Role: LEADER
I20260618 23:30:08.272842 27513 consensus_queue.cc:237] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 492d689d7b0449af92f2d448e8d3051a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492d689d7b0449af92f2d448e8d3051a" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42005 } } peers { permanent_uuid: "3336603f41764295a511dbb65169c654" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 41325 } } peers { permanent_uuid: "d9e7d40e8f1c4ff59a823315d295495c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 39055 } }
I20260618 23:30:08.273185 27176 raft_consensus.cc:2454] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P d9e7d40e8f1c4ff59a823315d295495c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 492d689d7b0449af92f2d448e8d3051a in term 1.
I20260618 23:30:08.273775 27030 catalog_manager.cc:5495] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 492d689d7b0449af92f2d448e8d3051a reported cstate change: term changed from 0 to 1, leader changed from <none> to 492d689d7b0449af92f2d448e8d3051a (127.8.39.130). New cstate: current_term: 1 leader_uuid: "492d689d7b0449af92f2d448e8d3051a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "492d689d7b0449af92f2d448e8d3051a" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 42005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3336603f41764295a511dbb65169c654" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 41325 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d9e7d40e8f1c4ff59a823315d295495c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 39055 } health_report { overall_health: UNKNOWN } } }
W20260618 23:30:08.280189 27223 tablet.cc:2369] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P d9e7d40e8f1c4ff59a823315d295495c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:30:08.309208 27502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:08.306588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35200 (local address 127.8.39.130:42005)
0618 23:30:08.306790 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:08.306792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:08.306802 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:30:08.306869 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:08.306870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:08.306891 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:08.306965 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:30:08.307136 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.307608 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.308374 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.308577 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.308655 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:08.308818 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:30:08.308920 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:30:08.309022 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:08.309040 (+ 18us) server_negotiation.cc:299] Negotiation successful
0618 23:30:08.309061 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169}
I20260618 23:30:08.311461 27176 raft_consensus.cc:1261] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P d9e7d40e8f1c4ff59a823315d295495c [term 1 FOLLOWER]: Refusing update from remote peer 492d689d7b0449af92f2d448e8d3051a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:30:08.311478 27444 raft_consensus.cc:1261] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 3336603f41764295a511dbb65169c654 [term 1 FOLLOWER]: Refusing update from remote peer 492d689d7b0449af92f2d448e8d3051a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:30:08.311833 27517 consensus_queue.cc:1035] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 492d689d7b0449af92f2d448e8d3051a [LEADER]: Connected to new peer: Peer: permanent_uuid: "3336603f41764295a511dbb65169c654" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 41325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:30:08.312041 27513 consensus_queue.cc:1035] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 492d689d7b0449af92f2d448e8d3051a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9e7d40e8f1c4ff59a823315d295495c" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 39055 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:30:08.315593 27521 mvcc.cc:205] Tried to move back new op lower bound from 7298356872439885824 to 7298356872285884416. Current Snapshot: MvccSnapshot[applied={T|T < 7298356872439885824 or (T in {7298356872439885824})}]
I20260618 23:30:08.316591 27030 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:30:08.316690 27030 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33642:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:30:08.317436 27030 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c2bc3eff370b481995f62fee3bf5c97c: Sending DeleteTablet for 3 replicas of tablet 0c2b1c7a082a4d97a2ca90e3ee22e8a7
I20260618 23:30:08.317735 27156 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c2b1c7a082a4d97a2ca90e3ee22e8a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:30:08 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:35434
I20260618 23:30:08.317780 27424 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c2b1c7a082a4d97a2ca90e3ee22e8a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:30:08 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:50514
I20260618 23:30:08.317935 27530 tablet_replica.cc:330] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P d9e7d40e8f1c4ff59a823315d295495c: stopping tablet replica
I20260618 23:30:08.318001 27531 tablet_replica.cc:330] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 3336603f41764295a511dbb65169c654: stopping tablet replica
I20260618 23:30:08.318028 27530 raft_consensus.cc:2229] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P d9e7d40e8f1c4ff59a823315d295495c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:30:08.318089 27531 raft_consensus.cc:2229] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 3336603f41764295a511dbb65169c654 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:30:08.318123 27530 raft_consensus.cc:2258] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P d9e7d40e8f1c4ff59a823315d295495c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:30:08.318181 27531 raft_consensus.cc:2258] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 3336603f41764295a511dbb65169c654 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:30:08.318249 27290 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c2b1c7a082a4d97a2ca90e3ee22e8a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:30:08 UTC) from {username='kudu', principal='kudu/127.8.39.190@KRBTEST.COM'} at 127.0.0.1:35188
I20260618 23:30:08.318614 27531 ts_tablet_manager.cc:1905] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 3336603f41764295a511dbb65169c654: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:30:08.318627 27530 ts_tablet_manager.cc:1905] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P d9e7d40e8f1c4ff59a823315d295495c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:30:08.318701 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 27093
I20260618 23:30:08.320776 27531 ts_tablet_manager.cc:1918] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 3336603f41764295a511dbb65169c654: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:30:08.320875 27531 log.cc:1196] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 3336603f41764295a511dbb65169c654: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781825053542692-8350-0/minicluster-data/ts-2/wal/wals/0c2b1c7a082a4d97a2ca90e3ee22e8a7
I20260618 23:30:08.321235 27531 ts_tablet_manager.cc:1939] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 3336603f41764295a511dbb65169c654: Deleting consensus metadata
I20260618 23:30:08.321622 27016 catalog_manager.cc:4841] TS 3336603f41764295a511dbb65169c654 (127.8.39.131:41325): tablet 0c2b1c7a082a4d97a2ca90e3ee22e8a7 (table test-table [id=847cb5c20ea44ca386ddc9de25d86172]) successfully deleted
I20260618 23:30:08.325631 27532 tablet_replica.cc:330] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 492d689d7b0449af92f2d448e8d3051a: stopping tablet replica
I20260618 23:30:08.325742 27532 raft_consensus.cc:2229] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 492d689d7b0449af92f2d448e8d3051a [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:30:08.325871 27532 raft_consensus.cc:2258] T 0c2b1c7a082a4d97a2ca90e3ee22e8a7 P 492d689d7b0449af92f2d448e8d3051a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260618 23:30:08.325974 27015 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.39.129:39055 (error 108)
I20260618 23:30:08.326074 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 27227
I20260618 23:30:08.328382 27504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:08.328119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:39055 (local address 127.0.0.1:35444)
0618 23:30:08.328295 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:08.328351 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.129:39055: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148}
W20260618 23:30:08.328528 27015 catalog_manager.cc:4584] TS d9e7d40e8f1c4ff59a823315d295495c (127.8.39.129:39055): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0c2b1c7a082a4d97a2ca90e3ee22e8a7: Network error: Client connection negotiation failed: client connection to 127.8.39.129:39055: connect: Connection refused (error 111)
I20260618 23:30:08.331688 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 27361
I20260618 23:30:08.331748 27504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:08.331634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:42005 (local address 127.0.0.1:35206)
0618 23:30:08.331680 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:08.331712 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.130:42005: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260618 23:30:08.336387 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 26999
2026-06-18T23:30:08Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2087 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-18T23:30:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:30:08Z Disabled control of system clock
I20260618 23:30:08.364364 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:45051
--webserver_interface=127.8.39.190
--webserver_port=0
--builtin_ntp_servers=127.8.39.148:33241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:45051
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 23:30:08.449815 27537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:08.450058 27537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:08.450080 27537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:08.451270 27537 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:30:08.451333 27537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:08.451349 27537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:08.451361 27537 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:30:08.451375 27537 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:30:08.452744 27537 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:33241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:45051
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:45051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:08.452998 27537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:08.453312 27537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:08.458067 27542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:08.458099 27543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:08.458067 27545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:08.458369 27537 server_base.cc:1034] running on GCE node
I20260618 23:30:08.458621 27537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:08.458896 27537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:08.460088 27537 hybrid_clock.cc:648] HybridClock initialized: now 1781825408460067 us; error 40 us; skew 500 ppm
I20260618 23:30:08.461560 27537 webserver.cc:460] Webserver started at http://127.8.39.190:35751/ using document root <none> and password file <none>
I20260618 23:30:08.461809 27537 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:08.461856 27537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:08.462041 27537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:08.463117 27537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "7ffff2852666435cb9589366d8f12b3f"
format_stamp: "Formatted at 2026-06-18 23:30:08 on dist-test-slave-47qt"
server_key: "e31bfcba9ad59e8ee504ed111ed06c71"
server_key_iv: "ff25866d96621cfaaacdfecc5b34786a"
server_key_version: "encryptionkey@0"
I20260618 23:30:08.463474 27537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "7ffff2852666435cb9589366d8f12b3f"
format_stamp: "Formatted at 2026-06-18 23:30:08 on dist-test-slave-47qt"
server_key: "e31bfcba9ad59e8ee504ed111ed06c71"
server_key_iv: "ff25866d96621cfaaacdfecc5b34786a"
server_key_version: "encryptionkey@0"
I20260618 23:30:08.464862 27537 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:30:08.465842 27551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:08.466128 27537 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:30:08.466234 27537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "7ffff2852666435cb9589366d8f12b3f"
format_stamp: "Formatted at 2026-06-18 23:30:08 on dist-test-slave-47qt"
server_key: "e31bfcba9ad59e8ee504ed111ed06c71"
server_key_iv: "ff25866d96621cfaaacdfecc5b34786a"
server_key_version: "encryptionkey@0"
I20260618 23:30:08.466320 27537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:08.477900 27537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:08.478515 27537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:08.478693 27537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:08.483558 27537 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:45051
I20260618 23:30:08.483660 27603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:45051 every 8 connection(s)
I20260618 23:30:08.484265 27537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:30:08.485803 27604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:08.488277 27604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ffff2852666435cb9589366d8f12b3f: Bootstrap starting.
I20260618 23:30:08.488953 27604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ffff2852666435cb9589366d8f12b3f: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:08.489053 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 27537
I20260618 23:30:08.489219 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:30:08.489319 27604 log.cc:824] T 00000000000000000000000000000000 P 7ffff2852666435cb9589366d8f12b3f: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:08.489454 8350 external_mini_cluster.cc:1428] Setting key c931d690b0ffb4a4cf2ec73b34fa465b
I20260618 23:30:08.490176 27604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ffff2852666435cb9589366d8f12b3f: No bootstrap required, opened a new log
I20260618 23:30:08.492580 27604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ffff2852666435cb9589366d8f12b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ffff2852666435cb9589366d8f12b3f" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 45051 } }
I20260618 23:30:08.492801 27604 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ffff2852666435cb9589366d8f12b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:08.492842 27604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ffff2852666435cb9589366d8f12b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ffff2852666435cb9589366d8f12b3f, State: Initialized, Role: FOLLOWER
I20260618 23:30:08.493045 27604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ffff2852666435cb9589366d8f12b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ffff2852666435cb9589366d8f12b3f" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 45051 } }
I20260618 23:30:08.493149 27604 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ffff2852666435cb9589366d8f12b3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:30:08.493189 27604 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ffff2852666435cb9589366d8f12b3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:30:08.493235 27604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ffff2852666435cb9589366d8f12b3f [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:08.494062 27604 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ffff2852666435cb9589366d8f12b3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ffff2852666435cb9589366d8f12b3f" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 45051 } }
I20260618 23:30:08.494206 27604 leader_election.cc:302] T 00000000000000000000000000000000 P 7ffff2852666435cb9589366d8f12b3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ffff2852666435cb9589366d8f12b3f; no voters:
I20260618 23:30:08.494410 27604 leader_election.cc:288] T 00000000000000000000000000000000 P 7ffff2852666435cb9589366d8f12b3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:30:08.494475 27609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ffff2852666435cb9589366d8f12b3f [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:30:08.494650 27609 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ffff2852666435cb9589366d8f12b3f [term 1 LEADER]: Becoming Leader. State: Replica: 7ffff2852666435cb9589366d8f12b3f, State: Running, Role: LEADER
I20260618 23:30:08.494697 27604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ffff2852666435cb9589366d8f12b3f [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:30:08.494737 27609 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ffff2852666435cb9589366d8f12b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ffff2852666435cb9589366d8f12b3f" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 45051 } }
I20260618 23:30:08.495067 27609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ffff2852666435cb9589366d8f12b3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ffff2852666435cb9589366d8f12b3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ffff2852666435cb9589366d8f12b3f" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 45051 } } }
I20260618 23:30:08.495219 27609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ffff2852666435cb9589366d8f12b3f [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:08.495391 27609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ffff2852666435cb9589366d8f12b3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ffff2852666435cb9589366d8f12b3f. Latest consensus state: current_term: 1 leader_uuid: "7ffff2852666435cb9589366d8f12b3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ffff2852666435cb9589366d8f12b3f" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 45051 } } }
I20260618 23:30:08.495604 27609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ffff2852666435cb9589366d8f12b3f [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:08.495934 27617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:30:08.496467 27617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:30:08.496915 27608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:08.490042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42410 (local address 127.8.39.190:45051)
0618 23:30:08.490260 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:08.490270 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:08.490308 (+ 38us) server_negotiation.cc:407] Connection header received
0618 23:30:08.490838 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:08.490847 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:08.490925 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:08.491112 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:08.491349 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.493413 (+ 2064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.494377 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.495053 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.495174 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:08.496135 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:08.496138 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:08.496139 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:08.496347 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:08.496390 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:08.496406 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:08.496476 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:08.496637 (+ 161us) server_negotiation.cc:299] Negotiation successful
0618 23:30:08.496718 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":78,"threads_started":1}
I20260618 23:30:08.498061 27617 catalog_manager.cc:1347] Generated new cluster ID: dd35c8475b8b4b2981a3c303efa4062f
I20260618 23:30:08.498118 27617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:30:08.510476 27617 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:30:08.511339 27617 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:30:08.516333 27617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ffff2852666435cb9589366d8f12b3f: Generated new TSK 0
I20260618 23:30:08.516911 27617 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:30:08.526260 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.129:0
--local_ip_for_outbound_sockets=127.8.39.129
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:45051
--builtin_ntp_servers=127.8.39.148:33241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 23:30:08.613925 27628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:08.614161 27628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:08.614197 27628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:08.615366 27628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:08.615440 27628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:08.615484 27628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.129
I20260618 23:30:08.616887 27628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:33241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.39.129
--webserver_port=0
--tserver_master_addrs=127.8.39.190:45051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.129
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:08.617209 27628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:08.617519 27628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:08.624686 27633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:08.624686 27634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:08.624686 27636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:08.624945 27628 server_base.cc:1034] running on GCE node
I20260618 23:30:08.625236 27628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:08.625564 27628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:08.626793 27628 hybrid_clock.cc:648] HybridClock initialized: now 1781825408626781 us; error 30 us; skew 500 ppm
I20260618 23:30:08.628659 27628 webserver.cc:460] Webserver started at http://127.8.39.129:42101/ using document root <none> and password file <none>
I20260618 23:30:08.628948 27628 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:08.628997 27628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:08.629204 27628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:08.630378 27628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-0/data/instance:
uuid: "fac6209c8dbe4fb3bcb9ecadaefced64"
format_stamp: "Formatted at 2026-06-18 23:30:08 on dist-test-slave-47qt"
server_key: "fc3ddf1e3ef2238e81bb9e48be34a8f9"
server_key_iv: "91da09f22411adfd9192c8b17decb46a"
server_key_version: "encryptionkey@0"
I20260618 23:30:08.630802 27628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance:
uuid: "fac6209c8dbe4fb3bcb9ecadaefced64"
format_stamp: "Formatted at 2026-06-18 23:30:08 on dist-test-slave-47qt"
server_key: "fc3ddf1e3ef2238e81bb9e48be34a8f9"
server_key_iv: "91da09f22411adfd9192c8b17decb46a"
server_key_version: "encryptionkey@0"
I20260618 23:30:08.632246 27628 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260618 23:30:08.633050 27642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:08.633256 27628 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:30:08.633352 27628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-0/wal
uuid: "fac6209c8dbe4fb3bcb9ecadaefced64"
format_stamp: "Formatted at 2026-06-18 23:30:08 on dist-test-slave-47qt"
server_key: "fc3ddf1e3ef2238e81bb9e48be34a8f9"
server_key_iv: "91da09f22411adfd9192c8b17decb46a"
server_key_version: "encryptionkey@0"
I20260618 23:30:08.633416 27628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:08.664865 27628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:08.665272 27628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:08.665397 27628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:08.665602 27628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:30:08.665951 27628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:30:08.665984 27628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:08.666066 27628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:30:08.666090 27628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:08.673616 27628 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.129:46847
I20260618 23:30:08.673660 27755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.129:46847 every 8 connection(s)
I20260618 23:30:08.674031 27628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-0/data/info.pb
I20260618 23:30:08.680141 27608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:08.674851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.129:56639 (local address 127.8.39.190:45051)
0618 23:30:08.674940 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:08.674943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:08.675127 (+ 184us) server_negotiation.cc:407] Connection header received
0618 23:30:08.675611 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:08.675612 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:08.675625 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:08.675675 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:08.676361 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.676856 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.678494 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.678716 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.678790 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:08.679772 (+ 982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:08.679774 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:08.679774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:08.679779 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:08.679793 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:08.679796 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:08.679843 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:08.680049 (+ 206us) server_negotiation.cc:299] Negotiation successful
0618 23:30:08.680072 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260618 23:30:08.680559 27758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:08.674819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:45051 (local address 127.8.39.129:56639)
0618 23:30:08.675009 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:08.675042 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:08.675519 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:08.675732 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:08.675735 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:08.675743 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:08.676266 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:08.676270 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.677087 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.677089 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:08.678339 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:08.678353 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.678863 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.678865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:08.679046 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:08.679625 (+ 579us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:30:08.679626 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:08.679656 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:30:08.679682 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:08.679913 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:08.679924 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:08.679927 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:30:08.680022 (+ 95us) client_negotiation.cc:240] Negotiation successful
0618 23:30:08.680154 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":105,"threads_started":1}
I20260618 23:30:08.681382 27756 heartbeater.cc:347] Connected to a master server at 127.8.39.190:45051
I20260618 23:30:08.681511 27756 heartbeater.cc:464] Registering TS with master...
I20260618 23:30:08.681735 27756 heartbeater.cc:511] Master 127.8.39.190:45051 requested a full tablet report, sending...
I20260618 23:30:08.682116 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 27628
I20260618 23:30:08.682227 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-0/wal/instance
I20260618 23:30:08.682384 27568 ts_manager.cc:194] Registered new tserver with Master: fac6209c8dbe4fb3bcb9ecadaefced64 (127.8.39.129:46847)
I20260618 23:30:08.682461 8350 external_mini_cluster.cc:1428] Setting key d617f53414d809a4ab91b462941e82d3
I20260618 23:30:08.683033 27568 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.39.129:56639
I20260618 23:30:08.683750 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.130:0
--local_ip_for_outbound_sockets=127.8.39.130
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:45051
--builtin_ntp_servers=127.8.39.148:33241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260618 23:30:08.768975 27759 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:08.769198 27759 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:08.769220 27759 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:08.770548 27759 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:08.770613 27759 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:08.770642 27759 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.130
I20260618 23:30:08.772133 27759 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:33241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.39.130
--webserver_port=0
--tserver_master_addrs=127.8.39.190:45051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27759
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.130
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:08.772382 27759 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:08.772680 27759 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:08.778208 27764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:08.778308 27765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:08.778337 27759 server_base.cc:1034] running on GCE node
W20260618 23:30:08.778239 27767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:08.778728 27759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:08.778923 27759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:08.780098 27759 hybrid_clock.cc:648] HybridClock initialized: now 1781825408780071 us; error 40 us; skew 500 ppm
I20260618 23:30:08.781553 27759 webserver.cc:460] Webserver started at http://127.8.39.130:38113/ using document root <none> and password file <none>
I20260618 23:30:08.781812 27759 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:08.781890 27759 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:08.782051 27759 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:08.783176 27759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-1/data/instance:
uuid: "2c233d73f735491da4d91a56ec6e6df5"
format_stamp: "Formatted at 2026-06-18 23:30:08 on dist-test-slave-47qt"
server_key: "ad24ab161ab9e68edaaf80e542d35525"
server_key_iv: "1194bc4dd35ef736ca518081d313573c"
server_key_version: "encryptionkey@0"
I20260618 23:30:08.783568 27759 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance:
uuid: "2c233d73f735491da4d91a56ec6e6df5"
format_stamp: "Formatted at 2026-06-18 23:30:08 on dist-test-slave-47qt"
server_key: "ad24ab161ab9e68edaaf80e542d35525"
server_key_iv: "1194bc4dd35ef736ca518081d313573c"
server_key_version: "encryptionkey@0"
I20260618 23:30:08.784973 27759 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:30:08.785789 27773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:08.785995 27759 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:30:08.786075 27759 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-1/wal
uuid: "2c233d73f735491da4d91a56ec6e6df5"
format_stamp: "Formatted at 2026-06-18 23:30:08 on dist-test-slave-47qt"
server_key: "ad24ab161ab9e68edaaf80e542d35525"
server_key_iv: "1194bc4dd35ef736ca518081d313573c"
server_key_version: "encryptionkey@0"
I20260618 23:30:08.786128 27759 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:08.800329 27759 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:08.800626 27759 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:08.800734 27759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:08.800964 27759 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:30:08.801367 27759 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:30:08.801407 27759 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:08.801431 27759 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:30:08.801443 27759 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:08.807225 27759 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.130:46061
I20260618 23:30:08.807307 27886 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.130:46061 every 8 connection(s)
I20260618 23:30:08.807646 27759 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-1/data/info.pb
I20260618 23:30:08.808923 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 27759
I20260618 23:30:08.809041 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-1/wal/instance
I20260618 23:30:08.809284 8350 external_mini_cluster.cc:1428] Setting key 870e813c3093cca4f085aacf68f97f0f
I20260618 23:30:08.810346 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.39.131:0
--local_ip_for_outbound_sockets=127.8.39.131
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:45051
--builtin_ntp_servers=127.8.39.148:33241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260618 23:30:08.814477 27608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:08.808489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.130:38549 (local address 127.8.39.190:45051)
0618 23:30:08.808582 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:08.808585 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:08.808680 (+ 95us) server_negotiation.cc:407] Connection header received
0618 23:30:08.809310 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:08.809311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:08.809325 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:08.809396 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:08.810224 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.811008 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.812437 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.812734 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.812798 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:08.814100 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:08.814102 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:08.814102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:08.814108 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:08.814127 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:08.814132 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:08.814199 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:08.814392 (+ 193us) server_negotiation.cc:299] Negotiation successful
0618 23:30:08.814416 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260618 23:30:08.814821 27889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:08.808385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:45051 (local address 127.8.39.130:38549)
0618 23:30:08.808565 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:08.808599 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:08.809212 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:08.809513 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:08.809518 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:08.809529 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:08.810139 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:08.810144 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.811119 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.811120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:08.812325 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:08.812328 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.812837 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.812838 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:08.812996 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:08.813784 (+ 788us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:30:08.813785 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:08.813848 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:30:08.813898 (+ 50us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:08.814270 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:08.814281 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:08.814285 (+ 4us) client_negotiation.cc:769] Sending connection context
0618 23:30:08.814396 (+ 111us) client_negotiation.cc:240] Negotiation successful
0618 23:30:08.814525 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":62,"threads_started":1}
I20260618 23:30:08.815454 27887 heartbeater.cc:347] Connected to a master server at 127.8.39.190:45051
I20260618 23:30:08.815562 27887 heartbeater.cc:464] Registering TS with master...
I20260618 23:30:08.815764 27887 heartbeater.cc:511] Master 127.8.39.190:45051 requested a full tablet report, sending...
I20260618 23:30:08.816174 27568 ts_manager.cc:194] Registered new tserver with Master: 2c233d73f735491da4d91a56ec6e6df5 (127.8.39.130:46061)
I20260618 23:30:08.816732 27568 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.39.130:38549
W20260618 23:30:08.900789 27890 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:08.901002 27890 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:08.901024 27890 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:08.902200 27890 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:08.902245 27890 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:08.902271 27890 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.39.131
I20260618 23:30:08.903610 27890 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:33241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.39.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.39.131
--webserver_port=0
--tserver_master_addrs=127.8.39.190:45051
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27890
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.39.131
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:08.903867 27890 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:08.904183 27890 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:08.909588 27896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:08.909549 27895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:08.909633 27890 server_base.cc:1034] running on GCE node
W20260618 23:30:08.909544 27898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:08.909957 27890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:08.910218 27890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:08.911422 27890 hybrid_clock.cc:648] HybridClock initialized: now 1781825408911404 us; error 45 us; skew 500 ppm
I20260618 23:30:08.912667 27890 webserver.cc:460] Webserver started at http://127.8.39.131:42587/ using document root <none> and password file <none>
I20260618 23:30:08.912911 27890 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:08.912977 27890 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:08.913100 27890 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:08.914191 27890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-2/data/instance:
uuid: "7eff1ee8f8184f829c4c69778d69ef55"
format_stamp: "Formatted at 2026-06-18 23:30:08 on dist-test-slave-47qt"
server_key: "27fc1ee41c886d1f74cbd5d545dc7203"
server_key_iv: "6001bcb8e8f226dd9967ddb38308353e"
server_key_version: "encryptionkey@0"
I20260618 23:30:08.914559 27890 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance:
uuid: "7eff1ee8f8184f829c4c69778d69ef55"
format_stamp: "Formatted at 2026-06-18 23:30:08 on dist-test-slave-47qt"
server_key: "27fc1ee41c886d1f74cbd5d545dc7203"
server_key_iv: "6001bcb8e8f226dd9967ddb38308353e"
server_key_version: "encryptionkey@0"
I20260618 23:30:08.916311 27890 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260618 23:30:08.917379 27904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:08.917632 27890 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:30:08.917744 27890 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-2/wal
uuid: "7eff1ee8f8184f829c4c69778d69ef55"
format_stamp: "Formatted at 2026-06-18 23:30:08 on dist-test-slave-47qt"
server_key: "27fc1ee41c886d1f74cbd5d545dc7203"
server_key_iv: "6001bcb8e8f226dd9967ddb38308353e"
server_key_version: "encryptionkey@0"
I20260618 23:30:08.917833 27890 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:08.928452 27890 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:08.928805 27890 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:08.928972 27890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:08.929275 27890 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260618 23:30:08.929634 27890 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260618 23:30:08.929695 27890 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:08.929740 27890 ts_tablet_manager.cc:610] Registered 0 tablets
I20260618 23:30:08.929766 27890 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:08.935787 27890 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.131:32857
I20260618 23:30:08.935830 28017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.131:32857 every 8 connection(s)
I20260618 23:30:08.936345 27890 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-2/data/info.pb
I20260618 23:30:08.936524 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 27890
I20260618 23:30:08.936637 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-2/wal/instance
I20260618 23:30:08.936813 8350 external_mini_cluster.cc:1428] Setting key 0dd634ce36a247355ee1ffff6ff65829
I20260618 23:30:08.941993 27608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:08.937236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:52409 (local address 127.8.39.190:45051)
0618 23:30:08.937288 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:08.937290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:08.937689 (+ 399us) server_negotiation.cc:407] Connection header received
0618 23:30:08.938266 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:08.938267 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:08.938277 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:08.938329 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:08.939012 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.939505 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.940548 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.940730 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.940790 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:08.941724 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:08.941725 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:08.941725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:08.941730 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:08.941743 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:08.941747 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:08.941792 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:08.941883 (+ 91us) server_negotiation.cc:299] Negotiation successful
0618 23:30:08.941905 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260618 23:30:08.942183 28020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:08.937325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:45051 (local address 127.8.39.131:52409)
0618 23:30:08.937555 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:08.937597 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:08.938160 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:08.938381 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:08.938385 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:08.938392 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:08.938895 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:08.938899 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.939632 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.939633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:08.940442 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:08.940445 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.940831 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.940832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:08.940952 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:08.941585 (+ 633us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0618 23:30:08.941586 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:08.941613 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0618 23:30:08.941636 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:08.941806 (+ 170us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:08.941810 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:08.941813 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:30:08.941866 (+ 53us) client_negotiation.cc:240] Negotiation successful
0618 23:30:08.941947 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":83,"threads_started":1}
I20260618 23:30:08.942657 28018 heartbeater.cc:347] Connected to a master server at 127.8.39.190:45051
I20260618 23:30:08.942739 28018 heartbeater.cc:464] Registering TS with master...
I20260618 23:30:08.942953 28018 heartbeater.cc:511] Master 127.8.39.190:45051 requested a full tablet report, sending...
I20260618 23:30:08.943511 27568 ts_manager.cc:194] Registered new tserver with Master: 7eff1ee8f8184f829c4c69778d69ef55 (127.8.39.131:32857)
I20260618 23:30:08.943905 27568 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.39.131:52409
I20260618 23:30:08.948719 8350 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260618 23:30:08.953749 27608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:08.950646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42426 (local address 127.8.39.190:45051)
0618 23:30:08.950709 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:08.950711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:08.950751 (+ 40us) server_negotiation.cc:407] Connection header received
0618 23:30:08.950792 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:08.950793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:08.950802 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:08.950865 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:08.951181 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.951663 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.952500 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.952690 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.952754 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:08.953526 (+ 772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:08.953527 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:08.953527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:08.953531 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:08.953544 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:08.953547 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:08.953599 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:08.953670 (+ 71us) server_negotiation.cc:299] Negotiation successful
0618 23:30:08.953690 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260618 23:30:08.955681 27568 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42426:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260618 23:30:08.957597 27568 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260618 23:30:08.964519 28030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:08.960400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44944 (local address 127.8.39.131:32857)
0618 23:30:08.961243 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:08.961246 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:08.961255 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:30:08.961292 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:08.961294 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:08.961309 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:08.961384 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:30:08.961530 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.962587 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.963596 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.964101 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.964203 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:08.964295 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:08.964385 (+ 90us) server_negotiation.cc:299] Negotiation successful
0618 23:30:08.964423 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":806,"thread_start_us":88,"threads_started":1}
I20260618 23:30:08.964519 28026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:08.960147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.131:32857 (local address 127.0.0.1:44944)
0618 23:30:08.960498 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:08.960508 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:08.960567 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:08.961399 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:08.961400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:08.961402 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:30:08.961436 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:08.961438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.962692 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.962694 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:08.963502 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:08.963504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.964207 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.964208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:08.964286 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:08.964298 (+ 12us) client_negotiation.cc:769] Sending connection context
0618 23:30:08.964372 (+ 74us) client_negotiation.cc:240] Negotiation successful
0618 23:30:08.964423 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":16,"spinlock_wait_cycles":8576,"thread_start_us":91,"threads_started":1}
I20260618 23:30:08.964871 28027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:08.960147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:46061 (local address 127.0.0.1:60472)
0618 23:30:08.960354 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:08.960394 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:08.960459 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:08.961155 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:08.961156 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:08.961157 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:30:08.961191 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:08.961192 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.962167 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.962168 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:08.963510 (+ 1342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:08.963513 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.964626 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.964627 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:08.964693 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:08.964696 (+ 3us) client_negotiation.cc:769] Sending connection context
0618 23:30:08.964748 (+ 52us) client_negotiation.cc:240] Negotiation successful
0618 23:30:08.964800 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"mutex_wait_us":1,"thread_start_us":81,"threads_started":1}
I20260618 23:30:08.964895 28031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:08.960587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51086 (local address 127.8.39.129:46847)
0618 23:30:08.960740 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:08.960745 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:08.960838 (+ 93us) server_negotiation.cc:407] Connection header received
0618 23:30:08.960915 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:08.960918 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:08.960935 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:08.961030 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:30:08.961232 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.962428 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.963647 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.964433 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.964508 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:08.964618 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:08.964784 (+ 166us) server_negotiation.cc:299] Negotiation successful
0618 23:30:08.964819 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":87,"threads_started":1}
I20260618 23:30:08.965051 28029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:08.960374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:46847 (local address 127.0.0.1:51086)
0618 23:30:08.960760 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:08.960769 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:08.960848 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:08.961042 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:08.961045 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:08.961047 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:30:08.961156 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:08.961158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.962529 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.962530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:08.963564 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:08.963566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.964647 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.964647 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:08.964694 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:08.964696 (+ 2us) client_negotiation.cc:769] Sending connection context
0618 23:30:08.964748 (+ 52us) client_negotiation.cc:240] Negotiation successful
0618 23:30:08.964997 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":370,"mutex_wait_us":203,"thread_start_us":68,"threads_started":1}
I20260618 23:30:08.965149 28028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:08.960309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60472 (local address 127.8.39.130:46061)
0618 23:30:08.960969 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:08.960973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:08.960982 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:30:08.961018 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:08.961020 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:08.961034 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:08.961107 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:30:08.961271 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.962073 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.963612 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:08.964439 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:08.964508 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:08.964587 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:08.964982 (+ 395us) server_negotiation.cc:299] Negotiation successful
0618 23:30:08.965024 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":96,"threads_started":1}
I20260618 23:30:08.965479 27952 tablet_service.cc:1467] Processing CreateTablet for tablet 0ee81ad31f6e49e8b1a7c7906fab4e06 (DEFAULT_TABLE table=test-table [id=c33e65054a9642958bf06f7c8d3eda54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:30:08.965658 27821 tablet_service.cc:1467] Processing CreateTablet for tablet 0ee81ad31f6e49e8b1a7c7906fab4e06 (DEFAULT_TABLE table=test-table [id=c33e65054a9642958bf06f7c8d3eda54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:30:08.965721 27690 tablet_service.cc:1467] Processing CreateTablet for tablet 0ee81ad31f6e49e8b1a7c7906fab4e06 (DEFAULT_TABLE table=test-table [id=c33e65054a9642958bf06f7c8d3eda54]), partition=RANGE (key) PARTITION UNBOUNDED
I20260618 23:30:08.965844 27952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ee81ad31f6e49e8b1a7c7906fab4e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:08.965969 27821 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ee81ad31f6e49e8b1a7c7906fab4e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:08.965987 27690 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ee81ad31f6e49e8b1a7c7906fab4e06. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:08.968427 28032 tablet_bootstrap.cc:492] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 2c233d73f735491da4d91a56ec6e6df5: Bootstrap starting.
I20260618 23:30:08.969045 28032 tablet_bootstrap.cc:654] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 2c233d73f735491da4d91a56ec6e6df5: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:08.969426 28032 log.cc:824] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 2c233d73f735491da4d91a56ec6e6df5: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:08.969547 28034 tablet_bootstrap.cc:492] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55: Bootstrap starting.
I20260618 23:30:08.969738 28033 tablet_bootstrap.cc:492] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P fac6209c8dbe4fb3bcb9ecadaefced64: Bootstrap starting.
I20260618 23:30:08.970208 28034 tablet_bootstrap.cc:654] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:08.970389 28032 tablet_bootstrap.cc:492] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 2c233d73f735491da4d91a56ec6e6df5: No bootstrap required, opened a new log
I20260618 23:30:08.970468 28033 tablet_bootstrap.cc:654] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P fac6209c8dbe4fb3bcb9ecadaefced64: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:08.970466 28032 ts_tablet_manager.cc:1397] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 2c233d73f735491da4d91a56ec6e6df5: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260618 23:30:08.970532 28034 log.cc:824] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:08.970789 28033 log.cc:824] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P fac6209c8dbe4fb3bcb9ecadaefced64: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:08.971216 28034 tablet_bootstrap.cc:492] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55: No bootstrap required, opened a new log
I20260618 23:30:08.971293 28034 ts_tablet_manager.cc:1397] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:30:08.971570 28033 tablet_bootstrap.cc:492] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P fac6209c8dbe4fb3bcb9ecadaefced64: No bootstrap required, opened a new log
I20260618 23:30:08.971622 28033 ts_tablet_manager.cc:1397] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P fac6209c8dbe4fb3bcb9ecadaefced64: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260618 23:30:08.972743 28032 raft_consensus.cc:348] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 2c233d73f735491da4d91a56ec6e6df5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c233d73f735491da4d91a56ec6e6df5" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46061 } } peers { permanent_uuid: "7eff1ee8f8184f829c4c69778d69ef55" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 32857 } } peers { permanent_uuid: "fac6209c8dbe4fb3bcb9ecadaefced64" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 46847 } }
I20260618 23:30:08.972873 28032 raft_consensus.cc:374] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 2c233d73f735491da4d91a56ec6e6df5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:08.972894 28032 raft_consensus.cc:729] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 2c233d73f735491da4d91a56ec6e6df5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c233d73f735491da4d91a56ec6e6df5, State: Initialized, Role: FOLLOWER
I20260618 23:30:08.973006 28032 consensus_queue.cc:260] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 2c233d73f735491da4d91a56ec6e6df5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c233d73f735491da4d91a56ec6e6df5" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46061 } } peers { permanent_uuid: "7eff1ee8f8184f829c4c69778d69ef55" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 32857 } } peers { permanent_uuid: "fac6209c8dbe4fb3bcb9ecadaefced64" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 46847 } }
I20260618 23:30:08.973265 28032 ts_tablet_manager.cc:1428] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 2c233d73f735491da4d91a56ec6e6df5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260618 23:30:08.973404 27887 heartbeater.cc:503] Master 127.8.39.190:45051 was elected leader, sending a full tablet report...
I20260618 23:30:08.973886 28034 raft_consensus.cc:348] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c233d73f735491da4d91a56ec6e6df5" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46061 } } peers { permanent_uuid: "7eff1ee8f8184f829c4c69778d69ef55" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 32857 } } peers { permanent_uuid: "fac6209c8dbe4fb3bcb9ecadaefced64" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 46847 } }
I20260618 23:30:08.974040 28034 raft_consensus.cc:374] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:08.974109 28034 raft_consensus.cc:729] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7eff1ee8f8184f829c4c69778d69ef55, State: Initialized, Role: FOLLOWER
I20260618 23:30:08.974143 28033 raft_consensus.cc:348] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P fac6209c8dbe4fb3bcb9ecadaefced64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c233d73f735491da4d91a56ec6e6df5" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46061 } } peers { permanent_uuid: "7eff1ee8f8184f829c4c69778d69ef55" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 32857 } } peers { permanent_uuid: "fac6209c8dbe4fb3bcb9ecadaefced64" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 46847 } }
I20260618 23:30:08.974228 28034 consensus_queue.cc:260] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c233d73f735491da4d91a56ec6e6df5" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46061 } } peers { permanent_uuid: "7eff1ee8f8184f829c4c69778d69ef55" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 32857 } } peers { permanent_uuid: "fac6209c8dbe4fb3bcb9ecadaefced64" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 46847 } }
I20260618 23:30:08.974277 28033 raft_consensus.cc:374] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P fac6209c8dbe4fb3bcb9ecadaefced64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:08.974318 28033 raft_consensus.cc:729] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P fac6209c8dbe4fb3bcb9ecadaefced64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fac6209c8dbe4fb3bcb9ecadaefced64, State: Initialized, Role: FOLLOWER
I20260618 23:30:08.974573 28034 ts_tablet_manager.cc:1428] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:30:08.974563 28033 consensus_queue.cc:260] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P fac6209c8dbe4fb3bcb9ecadaefced64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c233d73f735491da4d91a56ec6e6df5" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46061 } } peers { permanent_uuid: "7eff1ee8f8184f829c4c69778d69ef55" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 32857 } } peers { permanent_uuid: "fac6209c8dbe4fb3bcb9ecadaefced64" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 46847 } }
I20260618 23:30:08.974623 28018 heartbeater.cc:503] Master 127.8.39.190:45051 was elected leader, sending a full tablet report...
I20260618 23:30:08.974807 28033 ts_tablet_manager.cc:1428] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P fac6209c8dbe4fb3bcb9ecadaefced64: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260618 23:30:08.974857 27756 heartbeater.cc:503] Master 127.8.39.190:45051 was elected leader, sending a full tablet report...
W20260618 23:30:09.058276 27888 tablet.cc:2369] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 2c233d73f735491da4d91a56ec6e6df5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 23:30:09.174809 27757 tablet.cc:2369] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P fac6209c8dbe4fb3bcb9ecadaefced64: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260618 23:30:09.187177 28019 tablet.cc:2369] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260618 23:30:09.248903 28039 raft_consensus.cc:482] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:30:09.249043 28039 raft_consensus.cc:504] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c233d73f735491da4d91a56ec6e6df5" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46061 } } peers { permanent_uuid: "7eff1ee8f8184f829c4c69778d69ef55" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 32857 } } peers { permanent_uuid: "fac6209c8dbe4fb3bcb9ecadaefced64" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 46847 } }
I20260618 23:30:09.249531 28039 leader_election.cc:288] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c233d73f735491da4d91a56ec6e6df5 (127.8.39.130:46061), fac6209c8dbe4fb3bcb9ecadaefced64 (127.8.39.129:46847)
I20260618 23:30:09.253562 28031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:09.249743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:56285 (local address 127.8.39.129:46847)
0618 23:30:09.249813 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:09.249815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:09.249999 (+ 184us) server_negotiation.cc:407] Connection header received
0618 23:30:09.250033 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:09.250034 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:09.250047 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:09.250135 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:30:09.250263 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:09.250958 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:09.252579 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:09.253162 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:09.253240 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:09.253248 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:09.253451 (+ 203us) server_negotiation.cc:299] Negotiation successful
0618 23:30:09.253488 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260618 23:30:09.253572 28028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:09.249832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.39.131:47277 (local address 127.8.39.130:46061)
0618 23:30:09.249880 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:09.249882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:09.249891 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:30:09.249925 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:09.249926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:09.249935 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:09.250018 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0618 23:30:09.250186 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:09.250986 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:09.252586 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:09.253171 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:09.253240 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:09.253248 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:09.253451 (+ 203us) server_negotiation.cc:299] Negotiation successful
0618 23:30:09.253490 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260618 23:30:09.253649 28020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:09.249676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.130:46061 (local address 127.8.39.131:47277)
0618 23:30:09.249802 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:09.249815 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:09.249864 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:09.250033 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:09.250034 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:09.250036 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:30:09.250113 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:09.250115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:09.251099 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:09.251100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:09.252468 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:09.252471 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:09.253260 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:09.253261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:09.253361 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:09.253368 (+ 7us) client_negotiation.cc:769] Sending connection context
0618 23:30:09.253431 (+ 63us) client_negotiation.cc:240] Negotiation successful
0618 23:30:09.253493 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":136}
I20260618 23:30:09.253657 28041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:09.249688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:46847 (local address 127.8.39.131:56285)
0618 23:30:09.249920 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:09.249930 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:09.249987 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:09.250143 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:09.250144 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:09.250146 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0618 23:30:09.250205 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:09.250206 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:09.251099 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:09.251100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:09.252468 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:09.252471 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:09.253261 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:09.253261 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:09.253361 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:09.253368 (+ 7us) client_negotiation.cc:769] Sending connection context
0618 23:30:09.253431 (+ 63us) client_negotiation.cc:240] Negotiation successful
0618 23:30:09.253486 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":53,"thread_start_us":146,"threads_started":1}
I20260618 23:30:09.254009 27841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ee81ad31f6e49e8b1a7c7906fab4e06" candidate_uuid: "7eff1ee8f8184f829c4c69778d69ef55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c233d73f735491da4d91a56ec6e6df5" is_pre_election: true
I20260618 23:30:09.254154 27841 raft_consensus.cc:2454] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 2c233d73f735491da4d91a56ec6e6df5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eff1ee8f8184f829c4c69778d69ef55 in term 0.
I20260618 23:30:09.254387 27905 leader_election.cc:302] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c233d73f735491da4d91a56ec6e6df5, 7eff1ee8f8184f829c4c69778d69ef55; no voters:
I20260618 23:30:09.254384 27710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ee81ad31f6e49e8b1a7c7906fab4e06" candidate_uuid: "7eff1ee8f8184f829c4c69778d69ef55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fac6209c8dbe4fb3bcb9ecadaefced64" is_pre_election: true
I20260618 23:30:09.254514 28039 raft_consensus.cc:2781] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260618 23:30:09.254536 27710 raft_consensus.cc:2454] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P fac6209c8dbe4fb3bcb9ecadaefced64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7eff1ee8f8184f829c4c69778d69ef55 in term 0.
I20260618 23:30:09.254582 28039 raft_consensus.cc:482] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260618 23:30:09.254618 28039 raft_consensus.cc:3037] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:09.255326 28039 raft_consensus.cc:504] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c233d73f735491da4d91a56ec6e6df5" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46061 } } peers { permanent_uuid: "7eff1ee8f8184f829c4c69778d69ef55" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 32857 } } peers { permanent_uuid: "fac6209c8dbe4fb3bcb9ecadaefced64" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 46847 } }
I20260618 23:30:09.255478 28039 leader_election.cc:288] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55 [CANDIDATE]: Term 1 election: Requested vote from peers 2c233d73f735491da4d91a56ec6e6df5 (127.8.39.130:46061), fac6209c8dbe4fb3bcb9ecadaefced64 (127.8.39.129:46847)
I20260618 23:30:09.255618 27710 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ee81ad31f6e49e8b1a7c7906fab4e06" candidate_uuid: "7eff1ee8f8184f829c4c69778d69ef55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fac6209c8dbe4fb3bcb9ecadaefced64"
I20260618 23:30:09.255699 27710 raft_consensus.cc:3037] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P fac6209c8dbe4fb3bcb9ecadaefced64 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:09.256342 27710 raft_consensus.cc:2454] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P fac6209c8dbe4fb3bcb9ecadaefced64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eff1ee8f8184f829c4c69778d69ef55 in term 1.
I20260618 23:30:09.256482 27841 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ee81ad31f6e49e8b1a7c7906fab4e06" candidate_uuid: "7eff1ee8f8184f829c4c69778d69ef55" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c233d73f735491da4d91a56ec6e6df5"
I20260618 23:30:09.256527 27906 leader_election.cc:302] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7eff1ee8f8184f829c4c69778d69ef55, fac6209c8dbe4fb3bcb9ecadaefced64; no voters:
I20260618 23:30:09.256572 27841 raft_consensus.cc:3037] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 2c233d73f735491da4d91a56ec6e6df5 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:09.256686 28039 raft_consensus.cc:2781] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:30:09.256835 28039 raft_consensus.cc:686] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55 [term 1 LEADER]: Becoming Leader. State: Replica: 7eff1ee8f8184f829c4c69778d69ef55, State: Running, Role: LEADER
I20260618 23:30:09.256915 28039 consensus_queue.cc:237] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c233d73f735491da4d91a56ec6e6df5" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46061 } } peers { permanent_uuid: "7eff1ee8f8184f829c4c69778d69ef55" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 32857 } } peers { permanent_uuid: "fac6209c8dbe4fb3bcb9ecadaefced64" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 46847 } }
I20260618 23:30:09.257536 27841 raft_consensus.cc:2454] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 2c233d73f735491da4d91a56ec6e6df5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7eff1ee8f8184f829c4c69778d69ef55 in term 1.
I20260618 23:30:09.257911 27568 catalog_manager.cc:5495] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7eff1ee8f8184f829c4c69778d69ef55 (127.8.39.131). New cstate: current_term: 1 leader_uuid: "7eff1ee8f8184f829c4c69778d69ef55" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c233d73f735491da4d91a56ec6e6df5" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46061 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7eff1ee8f8184f829c4c69778d69ef55" member_type: VOTER last_known_addr { host: "127.8.39.131" port: 32857 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fac6209c8dbe4fb3bcb9ecadaefced64" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 46847 } health_report { overall_health: UNKNOWN } } }
I20260618 23:30:09.324371 28030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:09.321907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44960 (local address 127.8.39.131:32857)
0618 23:30:09.321958 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:09.321960 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:09.321971 (+ 11us) server_negotiation.cc:407] Connection header received
0618 23:30:09.322073 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:09.322074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:09.322092 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:09.322173 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:30:09.322325 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:09.322799 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:09.323564 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:09.323757 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:09.323832 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:09.324005 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:30:09.324113 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:30:09.324213 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:09.324233 (+ 20us) server_negotiation.cc:299] Negotiation successful
0618 23:30:09.324250 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260618 23:30:09.326361 27710 raft_consensus.cc:1261] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P fac6209c8dbe4fb3bcb9ecadaefced64 [term 1 FOLLOWER]: Refusing update from remote peer 7eff1ee8f8184f829c4c69778d69ef55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:30:09.326363 27841 raft_consensus.cc:1261] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 2c233d73f735491da4d91a56ec6e6df5 [term 1 FOLLOWER]: Refusing update from remote peer 7eff1ee8f8184f829c4c69778d69ef55: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:30:09.326718 28039 consensus_queue.cc:1035] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c233d73f735491da4d91a56ec6e6df5" member_type: VOTER last_known_addr { host: "127.8.39.130" port: 46061 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:30:09.326964 28042 consensus_queue.cc:1035] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fac6209c8dbe4fb3bcb9ecadaefced64" member_type: VOTER last_known_addr { host: "127.8.39.129" port: 46847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:30:09.329651 28045 mvcc.cc:205] Tried to move back new op lower bound from 7298356876597460992 to 7298356876316733440. Current Snapshot: MvccSnapshot[applied={T|T < 7298356876597460992}]
I20260618 23:30:09.329905 28046 mvcc.cc:205] Tried to move back new op lower bound from 7298356876597460992 to 7298356876316733440. Current Snapshot: MvccSnapshot[applied={T|T < 7298356876597460992}]
I20260618 23:30:09.333319 27568 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:30:09.333432 27568 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42426:
table { table_name: "test-table" } modify_external_catalogs: true
I20260618 23:30:09.334426 27568 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ffff2852666435cb9589366d8f12b3f: Sending DeleteTablet for 3 replicas of tablet 0ee81ad31f6e49e8b1a7c7906fab4e06
I20260618 23:30:09.334838 27952 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ee81ad31f6e49e8b1a7c7906fab4e06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:30:09 UTC) from {username='slave'} at 127.0.0.1:44944
I20260618 23:30:09.334869 27690 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ee81ad31f6e49e8b1a7c7906fab4e06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:30:09 UTC) from {username='slave'} at 127.0.0.1:51086
I20260618 23:30:09.334862 27821 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ee81ad31f6e49e8b1a7c7906fab4e06 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-18 23:30:09 UTC) from {username='slave'} at 127.0.0.1:60472
I20260618 23:30:09.335110 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 27628
I20260618 23:30:09.335213 28056 tablet_replica.cc:330] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55: stopping tablet replica
I20260618 23:30:09.335306 28056 raft_consensus.cc:2229] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55 [term 1 LEADER]: Raft consensus shutting down.
I20260618 23:30:09.335425 28056 raft_consensus.cc:2258] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:30:09.335898 28056 ts_tablet_manager.cc:1905] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:30:09.337276 28058 tablet_replica.cc:330] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 2c233d73f735491da4d91a56ec6e6df5: stopping tablet replica
I20260618 23:30:09.337386 28058 raft_consensus.cc:2229] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 2c233d73f735491da4d91a56ec6e6df5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260618 23:30:09.337489 28058 raft_consensus.cc:2258] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 2c233d73f735491da4d91a56ec6e6df5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260618 23:30:09.337986 28058 ts_tablet_manager.cc:1905] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 2c233d73f735491da4d91a56ec6e6df5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260618 23:30:09.338822 28056 ts_tablet_manager.cc:1918] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:30:09.338893 28056 log.cc:1196] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-2/wal/wals/0ee81ad31f6e49e8b1a7c7906fab4e06
I20260618 23:30:09.339157 28056 ts_tablet_manager.cc:1939] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 7eff1ee8f8184f829c4c69778d69ef55: Deleting consensus metadata
I20260618 23:30:09.339524 28058 ts_tablet_manager.cc:1918] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 2c233d73f735491da4d91a56ec6e6df5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260618 23:30:09.339584 28058 log.cc:1196] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 2c233d73f735491da4d91a56ec6e6df5: Deleting WAL directory at /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781825053542692-8350-0/minicluster-data/ts-1/wal/wals/0ee81ad31f6e49e8b1a7c7906fab4e06
I20260618 23:30:09.339790 27553 catalog_manager.cc:4841] TS 7eff1ee8f8184f829c4c69778d69ef55 (127.8.39.131:32857): tablet 0ee81ad31f6e49e8b1a7c7906fab4e06 (table test-table [id=c33e65054a9642958bf06f7c8d3eda54]) successfully deleted
I20260618 23:30:09.339841 28058 ts_tablet_manager.cc:1939] T 0ee81ad31f6e49e8b1a7c7906fab4e06 P 2c233d73f735491da4d91a56ec6e6df5: Deleting consensus metadata
I20260618 23:30:09.340279 27552 catalog_manager.cc:4841] TS 2c233d73f735491da4d91a56ec6e6df5 (127.8.39.130:46061): tablet 0ee81ad31f6e49e8b1a7c7906fab4e06 (table test-table [id=c33e65054a9642958bf06f7c8d3eda54]) successfully deleted
W20260618 23:30:09.343905 27553 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.39.129:46847 (error 108)
I20260618 23:30:09.344173 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 27759
I20260618 23:30:09.345665 28029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:09.344903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.129:46847 (local address 127.0.0.1:51098)
0618 23:30:09.345589 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:09.345634 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.129:46847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":665}
W20260618 23:30:09.345906 27553 catalog_manager.cc:4584] TS fac6209c8dbe4fb3bcb9ecadaefced64 (127.8.39.129:46847): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0ee81ad31f6e49e8b1a7c7906fab4e06: Network error: Client connection negotiation failed: client connection to 127.8.39.129:46847: connect: Connection refused (error 111)
I20260618 23:30:09.349637 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 27890
I20260618 23:30:09.355507 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 27537
2026-06-18T23:30:09Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1019 ms)
[----------] 6 tests from EncryptionPolicyTest (9179 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:30:09 dist-test-slave-47qt krb5kdc[28062](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:30:09 dist-test-slave-47qt krb5kdc[28062](info): set up 2 sockets
Jun 18 23:30:09 dist-test-slave-47qt krb5kdc[28062](info): commencing operation
krb5kdc: starting...
W20260618 23:30:10.429189 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:30:10 dist-test-slave-47qt krb5kdc[28062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:30:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:30:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 23:30:10.564359 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44177
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:30:10.652997 28075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:10.653254 28075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:10.653283 28075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:10.654697 28075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:30:10.654754 28075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:10.654770 28075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:10.654783 28075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:30:10.654796 28075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:30:10.656466 28075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44177
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28075
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:10.656720 28075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:10.656975 28075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 23:30:10.662340 28075 server_base.cc:1034] running on GCE node
W20260618 23:30:10.662292 28080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:10.662276 28083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:10.662308 28081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:10.662788 28075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:10.663107 28075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:10.664281 28075 hybrid_clock.cc:648] HybridClock initialized: now 1781825410664261 us; error 42 us; skew 500 ppm
Jun 18 23:30:10 dist-test-slave-47qt krb5kdc[28062](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825410, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:10.667726 28075 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 23:30:10.668242 28075 webserver.cc:460] Webserver started at http://127.0.0.1:37249/ using document root <none> and password file <none>
I20260618 23:30:10.668480 28075 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:10.668547 28075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:10.668701 28075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:10.669837 28075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "312ef69ce7b9431380e23a1e6df80825"
format_stamp: "Formatted at 2026-06-18 23:30:10 on dist-test-slave-47qt"
server_key: "c9ba6218556bb2d76a8735f72c36dc01"
server_key_iv: "686da4cb3e127592fe199d48f5c1ee05"
server_key_version: "encryptionkey@0"
I20260618 23:30:10.670233 28075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "312ef69ce7b9431380e23a1e6df80825"
format_stamp: "Formatted at 2026-06-18 23:30:10 on dist-test-slave-47qt"
server_key: "c9ba6218556bb2d76a8735f72c36dc01"
server_key_iv: "686da4cb3e127592fe199d48f5c1ee05"
server_key_version: "encryptionkey@0"
I20260618 23:30:10.671792 28075 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:30:10.672652 28090 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:10.672996 28075 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:30:10.673192 28075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "312ef69ce7b9431380e23a1e6df80825"
format_stamp: "Formatted at 2026-06-18 23:30:10 on dist-test-slave-47qt"
server_key: "c9ba6218556bb2d76a8735f72c36dc01"
server_key_iv: "686da4cb3e127592fe199d48f5c1ee05"
server_key_version: "encryptionkey@0"
I20260618 23:30:10.673275 28075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:10.699064 28075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:10.701901 28075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:10.702100 28075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:10.706596 28075 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44177
I20260618 23:30:10.706632 28142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44177 every 8 connection(s)
I20260618 23:30:10.706997 28075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:30:10.708397 28143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:10.710748 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 28075
I20260618 23:30:10.710793 28143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 312ef69ce7b9431380e23a1e6df80825: Bootstrap starting.
I20260618 23:30:10.710883 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:30:10.711071 8350 external_mini_cluster.cc:1428] Setting key e39048327f4198fd40ad1fdd061cf62b
I20260618 23:30:10.711453 28143 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 312ef69ce7b9431380e23a1e6df80825: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:10.711777 28143 log.cc:824] T 00000000000000000000000000000000 P 312ef69ce7b9431380e23a1e6df80825: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:10.712554 28143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 312ef69ce7b9431380e23a1e6df80825: No bootstrap required, opened a new log
I20260618 23:30:10.714810 28143 raft_consensus.cc:348] T 00000000000000000000000000000000 P 312ef69ce7b9431380e23a1e6df80825 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "312ef69ce7b9431380e23a1e6df80825" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44177 } }
I20260618 23:30:10.714983 28143 raft_consensus.cc:374] T 00000000000000000000000000000000 P 312ef69ce7b9431380e23a1e6df80825 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:10.715018 28143 raft_consensus.cc:729] T 00000000000000000000000000000000 P 312ef69ce7b9431380e23a1e6df80825 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 312ef69ce7b9431380e23a1e6df80825, State: Initialized, Role: FOLLOWER
I20260618 23:30:10.715147 28143 consensus_queue.cc:260] T 00000000000000000000000000000000 P 312ef69ce7b9431380e23a1e6df80825 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "312ef69ce7b9431380e23a1e6df80825" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44177 } }
I20260618 23:30:10.715217 28143 raft_consensus.cc:388] T 00000000000000000000000000000000 P 312ef69ce7b9431380e23a1e6df80825 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:30:10.715245 28143 raft_consensus.cc:482] T 00000000000000000000000000000000 P 312ef69ce7b9431380e23a1e6df80825 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:30:10.715286 28143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 312ef69ce7b9431380e23a1e6df80825 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:10.715886 28143 raft_consensus.cc:504] T 00000000000000000000000000000000 P 312ef69ce7b9431380e23a1e6df80825 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "312ef69ce7b9431380e23a1e6df80825" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44177 } }
I20260618 23:30:10.715969 28143 leader_election.cc:302] T 00000000000000000000000000000000 P 312ef69ce7b9431380e23a1e6df80825 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 312ef69ce7b9431380e23a1e6df80825; no voters:
I20260618 23:30:10.716153 28143 leader_election.cc:288] T 00000000000000000000000000000000 P 312ef69ce7b9431380e23a1e6df80825 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:30:10.716254 28148 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 312ef69ce7b9431380e23a1e6df80825 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:30:10.716418 28143 sys_catalog.cc:564] T 00000000000000000000000000000000 P 312ef69ce7b9431380e23a1e6df80825 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:30:10.716440 28148 raft_consensus.cc:686] T 00000000000000000000000000000000 P 312ef69ce7b9431380e23a1e6df80825 [term 1 LEADER]: Becoming Leader. State: Replica: 312ef69ce7b9431380e23a1e6df80825, State: Running, Role: LEADER
I20260618 23:30:10.716615 28148 consensus_queue.cc:237] T 00000000000000000000000000000000 P 312ef69ce7b9431380e23a1e6df80825 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "312ef69ce7b9431380e23a1e6df80825" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44177 } }
I20260618 23:30:10.717185 28150 sys_catalog.cc:455] T 00000000000000000000000000000000 P 312ef69ce7b9431380e23a1e6df80825 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 312ef69ce7b9431380e23a1e6df80825. Latest consensus state: current_term: 1 leader_uuid: "312ef69ce7b9431380e23a1e6df80825" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "312ef69ce7b9431380e23a1e6df80825" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44177 } } }
I20260618 23:30:10.717285 28150 sys_catalog.cc:458] T 00000000000000000000000000000000 P 312ef69ce7b9431380e23a1e6df80825 [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:10.717171 28149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 312ef69ce7b9431380e23a1e6df80825 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "312ef69ce7b9431380e23a1e6df80825" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "312ef69ce7b9431380e23a1e6df80825" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44177 } } }
I20260618 23:30:10.717442 28149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 312ef69ce7b9431380e23a1e6df80825 [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:10.718250 28156 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 18 23:30:10 dist-test-slave-47qt krb5kdc[28062](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825410, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260618 23:30:10.718792 28156 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:30:10.720819 28156 catalog_manager.cc:1347] Generated new cluster ID: 0b54c26c880f40048e26d4a2d0688111
I20260618 23:30:10.720876 28156 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:30:10.723203 28146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:10.711606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52441 (local address 127.0.0.1:44177)
0618 23:30:10.711820 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:10.711828 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:10.711879 (+ 51us) server_negotiation.cc:407] Connection header received
0618 23:30:10.712480 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:10.712487 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:10.712537 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:10.712710 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:10.713560 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:10.714515 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:10.715548 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:10.715993 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:10.716103 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:10.719173 (+ 3070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:10.719178 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:10.719178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:10.719193 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:10.721224 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:10.721705 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:10.721708 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:10.721709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:10.721842 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:10.722116 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:10.722117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:10.722117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:10.722444 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:10.722627 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:10.722931 (+ 304us) server_negotiation.cc:299] Negotiation successful
0618 23:30:10.723032 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":88,"threads_started":1}
I20260618 23:30:10.726526 8350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:30:10.732244 28156 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:30:10.732916 28156 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:30:10.734736 28146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:10.727855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32967 (local address 127.0.0.1:44177)
0618 23:30:10.727889 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:10.727892 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:10.728054 (+ 162us) server_negotiation.cc:407] Connection header received
0618 23:30:10.728098 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:10.728099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:10.728108 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:10.728162 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:10.728670 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:10.729409 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:10.730289 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:10.730554 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:10.730652 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:10.732129 (+ 1477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:10.732131 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:10.732137 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:10.732146 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:10.733600 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:10.733970 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:10.733971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:10.733972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:10.734025 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:10.734244 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:10.734245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:10.734245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:10.734350 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:10.734460 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:10.734571 (+ 111us) server_negotiation.cc:299] Negotiation successful
0618 23:30:10.734634 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260618 23:30:10.749670 28156 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 312ef69ce7b9431380e23a1e6df80825: Generated new TSK 0
I20260618 23:30:10.749908 28156 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:30:10.753595 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 28075
W20260618 23:30:10.757746 28060 connection.cc:535] client connection to 127.0.0.1:44177 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44177): Connection reset by peer (error 104)
2026-06-18T23:30:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1395 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:30:10 dist-test-slave-47qt krb5kdc[28174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:30:10 dist-test-slave-47qt krb5kdc[28174](info): set up 2 sockets
Jun 18 23:30:10 dist-test-slave-47qt krb5kdc[28174](info): commencing operation
krb5kdc: starting...
W20260618 23:30:11.841454 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.070s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:30:11 dist-test-slave-47qt krb5kdc[28174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825411, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:30:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:30:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 23:30:11.973661 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33235
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33235
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:30:12.057205 28187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:12.057489 28187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:12.057533 28187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:12.058691 28187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:30:12.058761 28187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:12.058787 28187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:12.058809 28187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:30:12.058838 28187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:30:12.060308 28187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33235
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33235
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28187
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:12.060588 28187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:12.060812 28187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:12.065778 28195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:12.065735 28192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:12.065752 28193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:12.066027 28187 server_base.cc:1034] running on GCE node
I20260618 23:30:12.066248 28187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:12.066488 28187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:12.067633 28187 hybrid_clock.cc:648] HybridClock initialized: now 1781825412067611 us; error 37 us; skew 500 ppm
Jun 18 23:30:12 dist-test-slave-47qt krb5kdc[28174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825412, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:12.070176 28187 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 23:30:12.070547 28187 webserver.cc:460] Webserver started at http://127.0.0.1:43129/ using document root <none> and password file <none>
I20260618 23:30:12.070899 28187 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:12.071009 28187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:12.071170 28187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:12.072239 28187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "b3cb52f3edcb40e0aeffaf30b6203ef0"
format_stamp: "Formatted at 2026-06-18 23:30:12 on dist-test-slave-47qt"
server_key: "b67af26c7ff241efc0c4e2adbbcce841"
server_key_iv: "0e70fa557b02f41bfbd19dbee9450dfb"
server_key_version: "encryptionkey@0"
I20260618 23:30:12.072600 28187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "b3cb52f3edcb40e0aeffaf30b6203ef0"
format_stamp: "Formatted at 2026-06-18 23:30:12 on dist-test-slave-47qt"
server_key: "b67af26c7ff241efc0c4e2adbbcce841"
server_key_iv: "0e70fa557b02f41bfbd19dbee9450dfb"
server_key_version: "encryptionkey@0"
I20260618 23:30:12.074056 28187 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:30:12.074865 28202 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:12.075062 28187 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:30:12.075170 28187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "b3cb52f3edcb40e0aeffaf30b6203ef0"
format_stamp: "Formatted at 2026-06-18 23:30:12 on dist-test-slave-47qt"
server_key: "b67af26c7ff241efc0c4e2adbbcce841"
server_key_iv: "0e70fa557b02f41bfbd19dbee9450dfb"
server_key_version: "encryptionkey@0"
I20260618 23:30:12.075274 28187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:12.094316 28187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:12.096719 28187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:12.096897 28187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:12.101732 28187 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33235
I20260618 23:30:12.101769 28254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33235 every 8 connection(s)
I20260618 23:30:12.102125 28187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:30:12.103152 28255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:12.105585 28255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3cb52f3edcb40e0aeffaf30b6203ef0: Bootstrap starting.
I20260618 23:30:12.106112 28255 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3cb52f3edcb40e0aeffaf30b6203ef0: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:12.106361 28255 log.cc:824] T 00000000000000000000000000000000 P b3cb52f3edcb40e0aeffaf30b6203ef0: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:12.107326 28255 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3cb52f3edcb40e0aeffaf30b6203ef0: No bootstrap required, opened a new log
I20260618 23:30:12.109349 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 28187
I20260618 23:30:12.109443 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:30:12.109615 8350 external_mini_cluster.cc:1428] Setting key 9c50d84655d86bc5eaeec88791e6c26b
I20260618 23:30:12.109941 28255 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3cb52f3edcb40e0aeffaf30b6203ef0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3cb52f3edcb40e0aeffaf30b6203ef0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33235 } }
I20260618 23:30:12.110190 28255 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3cb52f3edcb40e0aeffaf30b6203ef0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:12.110304 28255 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3cb52f3edcb40e0aeffaf30b6203ef0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3cb52f3edcb40e0aeffaf30b6203ef0, State: Initialized, Role: FOLLOWER
I20260618 23:30:12.110463 28255 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3cb52f3edcb40e0aeffaf30b6203ef0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3cb52f3edcb40e0aeffaf30b6203ef0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33235 } }
I20260618 23:30:12.110576 28255 raft_consensus.cc:388] T 00000000000000000000000000000000 P b3cb52f3edcb40e0aeffaf30b6203ef0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:30:12.110623 28255 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3cb52f3edcb40e0aeffaf30b6203ef0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:30:12.110680 28255 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3cb52f3edcb40e0aeffaf30b6203ef0 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:12.111874 28255 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3cb52f3edcb40e0aeffaf30b6203ef0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3cb52f3edcb40e0aeffaf30b6203ef0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33235 } }
I20260618 23:30:12.112010 28255 leader_election.cc:302] T 00000000000000000000000000000000 P b3cb52f3edcb40e0aeffaf30b6203ef0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b3cb52f3edcb40e0aeffaf30b6203ef0; no voters:
I20260618 23:30:12.112238 28255 leader_election.cc:288] T 00000000000000000000000000000000 P b3cb52f3edcb40e0aeffaf30b6203ef0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:30:12.112284 28260 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3cb52f3edcb40e0aeffaf30b6203ef0 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:30:12.112489 28260 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3cb52f3edcb40e0aeffaf30b6203ef0 [term 1 LEADER]: Becoming Leader. State: Replica: b3cb52f3edcb40e0aeffaf30b6203ef0, State: Running, Role: LEADER
I20260618 23:30:12.112537 28255 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3cb52f3edcb40e0aeffaf30b6203ef0 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:30:12.112625 28260 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3cb52f3edcb40e0aeffaf30b6203ef0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3cb52f3edcb40e0aeffaf30b6203ef0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33235 } }
I20260618 23:30:12.113029 28260 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3cb52f3edcb40e0aeffaf30b6203ef0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3cb52f3edcb40e0aeffaf30b6203ef0. Latest consensus state: current_term: 1 leader_uuid: "b3cb52f3edcb40e0aeffaf30b6203ef0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3cb52f3edcb40e0aeffaf30b6203ef0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33235 } } }
I20260618 23:30:12.113036 28262 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3cb52f3edcb40e0aeffaf30b6203ef0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b3cb52f3edcb40e0aeffaf30b6203ef0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b3cb52f3edcb40e0aeffaf30b6203ef0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33235 } } }
I20260618 23:30:12.113198 28260 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3cb52f3edcb40e0aeffaf30b6203ef0 [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:12.113205 28262 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3cb52f3edcb40e0aeffaf30b6203ef0 [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:12.113478 28269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:30:12.114166 28269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:30:12.115998 28269 catalog_manager.cc:1347] Generated new cluster ID: dfa68a3a22b94a8a8a89d1c917085d8b
I20260618 23:30:12.116052 28269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:30:12.116051 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:12.110239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44787 (local address 127.0.0.1:33235)
0618 23:30:12.110476 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:12.110484 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:12.110510 (+ 26us) server_negotiation.cc:407] Connection header received
0618 23:30:12.110628 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:12.110635 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:12.110698 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:12.110877 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:12.111098 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:12.112031 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:12.113358 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:12.114199 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:12.114299 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:12.115285 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:12.115288 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:12.115289 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:12.115545 (+ 256us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:12.115584 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:12.115599 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:12.115714 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:12.115802 (+ 88us) server_negotiation.cc:299] Negotiation successful
0618 23:30:12.115831 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":144,"threads_started":1}
I20260618 23:30:12.117077 8350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:30:12.122581 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:12.118630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39531 (local address 127.0.0.1:33235)
0618 23:30:12.118662 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:12.118663 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:12.118813 (+ 150us) server_negotiation.cc:407] Connection header received
0618 23:30:12.118841 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:12.118842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:12.118854 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:12.118920 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:12.119069 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:12.119783 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:12.121111 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:12.121446 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:12.121508 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:12.122301 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:12.122302 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:12.122303 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:12.122307 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:12.122322 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:12.122326 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:12.122382 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:12.122486 (+ 104us) server_negotiation.cc:299] Negotiation successful
0618 23:30:12.122500 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260618 23:30:12.125000 28269 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:30:12.125871 28269 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:30:12.133932 28269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b3cb52f3edcb40e0aeffaf30b6203ef0: Generated new TSK 0
I20260618 23:30:12.134128 28269 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:30:12.138062 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 28187
W20260618 23:30:12.143167 28172 connection.cc:535] client connection to 127.0.0.1:33235 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33235): Connection reset by peer (error 104)
2026-06-18T23:30:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1385 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:30:12 dist-test-slave-47qt krb5kdc[28286](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:30:12 dist-test-slave-47qt krb5kdc[28286](info): set up 2 sockets
Jun 18 23:30:12 dist-test-slave-47qt krb5kdc[28286](info): commencing operation
krb5kdc: starting...
W20260618 23:30:13.189350 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:30:13 dist-test-slave-47qt krb5kdc[28286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:30:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:30:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 23:30:13.312629 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43671
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:30:13.395982 28299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:13.396246 28299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:13.396287 28299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:13.397455 28299 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:30:13.397531 28299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:13.397557 28299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:13.397584 28299 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:30:13.397614 28299 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:30:13.399085 28299 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43671
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:13.399513 28299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:13.399771 28299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:13.405828 28304 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:13.405805 28307 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:13.405843 28299 server_base.cc:1034] running on GCE node
W20260618 23:30:13.405809 28305 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:13.406347 28299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:13.406605 28299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:13.407788 28299 hybrid_clock.cc:648] HybridClock initialized: now 1781825413407750 us; error 65 us; skew 500 ppm
Jun 18 23:30:13 dist-test-slave-47qt krb5kdc[28286](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825413, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:13.410414 28299 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 23:30:13.410898 28299 webserver.cc:460] Webserver started at http://127.0.0.1:40945/ using document root <none> and password file <none>
I20260618 23:30:13.411161 28299 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:13.411228 28299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:13.411374 28299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:13.412431 28299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "ec5b638ceb3e480d88461ce7b63249e6"
format_stamp: "Formatted at 2026-06-18 23:30:13 on dist-test-slave-47qt"
server_key: "f23d7b42e117575ed73cbf99e1b0a92b"
server_key_iv: "2157786adafe9db2b76ec42f3375daf7"
server_key_version: "encryptionkey@0"
I20260618 23:30:13.412793 28299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "ec5b638ceb3e480d88461ce7b63249e6"
format_stamp: "Formatted at 2026-06-18 23:30:13 on dist-test-slave-47qt"
server_key: "f23d7b42e117575ed73cbf99e1b0a92b"
server_key_iv: "2157786adafe9db2b76ec42f3375daf7"
server_key_version: "encryptionkey@0"
I20260618 23:30:13.414364 28299 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:30:13.415107 28314 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:13.415274 28299 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260618 23:30:13.415378 28299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "ec5b638ceb3e480d88461ce7b63249e6"
format_stamp: "Formatted at 2026-06-18 23:30:13 on dist-test-slave-47qt"
server_key: "f23d7b42e117575ed73cbf99e1b0a92b"
server_key_iv: "2157786adafe9db2b76ec42f3375daf7"
server_key_version: "encryptionkey@0"
I20260618 23:30:13.415452 28299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:13.419173 28299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:13.421849 28299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:13.422004 28299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:13.425838 28299 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43671
I20260618 23:30:13.426002 28366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43671 every 8 connection(s)
I20260618 23:30:13.426242 28299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:30:13.427233 28367 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:13.427510 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 28299
I20260618 23:30:13.427587 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:30:13.427722 8350 external_mini_cluster.cc:1428] Setting key d8175168cb3d7d74fd1695b3cb9a8301
I20260618 23:30:13.430866 28370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:13.428617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53095 (local address 127.0.0.1:43671)
0618 23:30:13.428820 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:13.428829 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:13.428864 (+ 35us) server_negotiation.cc:407] Connection header received
0618 23:30:13.429594 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:13.429602 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:13.429634 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:13.429791 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:13.430458 (+ 667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:13.430462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:13.430462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:13.430466 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:13.430510 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:13.430524 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:13.430598 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:13.430651 (+ 53us) server_negotiation.cc:299] Negotiation successful
0618 23:30:13.430691 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":106,"threads_started":1}
I20260618 23:30:13.431887 28367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec5b638ceb3e480d88461ce7b63249e6: Bootstrap starting.
I20260618 23:30:13.431968 8350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:30:13.432742 28367 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec5b638ceb3e480d88461ce7b63249e6: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:13.433432 28367 log.cc:824] T 00000000000000000000000000000000 P ec5b638ceb3e480d88461ce7b63249e6: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:13.434460 28367 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec5b638ceb3e480d88461ce7b63249e6: No bootstrap required, opened a new log
I20260618 23:30:13.435428 28370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:13.434249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36055 (local address 127.0.0.1:43671)
0618 23:30:13.434309 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:13.434311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:13.434438 (+ 127us) server_negotiation.cc:407] Connection header received
0618 23:30:13.434504 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:13.434505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:13.434530 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:13.434587 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:13.435223 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:13.435224 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:13.435225 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:13.435229 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:13.435242 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:13.435246 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:13.435290 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:13.435343 (+ 53us) server_negotiation.cc:299] Negotiation successful
0618 23:30:13.435357 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260618 23:30:13.436475 28367 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec5b638ceb3e480d88461ce7b63249e6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec5b638ceb3e480d88461ce7b63249e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43671 } }
I20260618 23:30:13.436623 28367 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec5b638ceb3e480d88461ce7b63249e6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:13.436647 28367 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec5b638ceb3e480d88461ce7b63249e6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec5b638ceb3e480d88461ce7b63249e6, State: Initialized, Role: FOLLOWER
I20260618 23:30:13.436762 28367 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec5b638ceb3e480d88461ce7b63249e6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec5b638ceb3e480d88461ce7b63249e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43671 } }
I20260618 23:30:13.436849 28367 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec5b638ceb3e480d88461ce7b63249e6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:30:13.436874 28367 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec5b638ceb3e480d88461ce7b63249e6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:30:13.436903 28367 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec5b638ceb3e480d88461ce7b63249e6 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:13.437525 28367 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec5b638ceb3e480d88461ce7b63249e6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec5b638ceb3e480d88461ce7b63249e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43671 } }
I20260618 23:30:13.437611 28367 leader_election.cc:302] T 00000000000000000000000000000000 P ec5b638ceb3e480d88461ce7b63249e6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec5b638ceb3e480d88461ce7b63249e6; no voters:
I20260618 23:30:13.437731 28367 leader_election.cc:288] T 00000000000000000000000000000000 P ec5b638ceb3e480d88461ce7b63249e6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:30:13.437817 28377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec5b638ceb3e480d88461ce7b63249e6 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:30:13.438002 28367 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec5b638ceb3e480d88461ce7b63249e6 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:30:13.438022 28377 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec5b638ceb3e480d88461ce7b63249e6 [term 1 LEADER]: Becoming Leader. State: Replica: ec5b638ceb3e480d88461ce7b63249e6, State: Running, Role: LEADER
I20260618 23:30:13.438108 28377 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec5b638ceb3e480d88461ce7b63249e6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec5b638ceb3e480d88461ce7b63249e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43671 } }
I20260618 23:30:13.438529 28379 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec5b638ceb3e480d88461ce7b63249e6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec5b638ceb3e480d88461ce7b63249e6. Latest consensus state: current_term: 1 leader_uuid: "ec5b638ceb3e480d88461ce7b63249e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec5b638ceb3e480d88461ce7b63249e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43671 } } }
I20260618 23:30:13.438514 28378 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec5b638ceb3e480d88461ce7b63249e6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec5b638ceb3e480d88461ce7b63249e6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec5b638ceb3e480d88461ce7b63249e6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43671 } } }
I20260618 23:30:13.438629 28379 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec5b638ceb3e480d88461ce7b63249e6 [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:13.438637 28378 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec5b638ceb3e480d88461ce7b63249e6 [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:13.438858 28386 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:30:13.439477 28386 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:30:13.441614 28386 catalog_manager.cc:1347] Generated new cluster ID: 48064c2b175847b99d6d4f9022b1d6f6
I20260618 23:30:13.441676 28386 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:30:13.485901 28386 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:30:13.486284 28386 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:30:13.496109 28386 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec5b638ceb3e480d88461ce7b63249e6: Generated new TSK 0
I20260618 23:30:13.496443 28386 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:30:13.507558 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 28299
2026-06-18T23:30:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1369 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:30:13 dist-test-slave-47qt krb5kdc[28398](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:30:13 dist-test-slave-47qt krb5kdc[28398](info): set up 2 sockets
Jun 18 23:30:13 dist-test-slave-47qt krb5kdc[28398](info): commencing operation
krb5kdc: starting...
W20260618 23:30:14.561180 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:30:14 dist-test-slave-47qt krb5kdc[28398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:30:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:30:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 23:30:14.690152 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36943
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36943
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:30:14.772307 28411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:14.772594 28411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:14.772639 28411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:14.773864 28411 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:30:14.773939 28411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:14.773963 28411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:14.773985 28411 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:30:14.774013 28411 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:30:14.775537 28411 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36943
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36943
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:14.775823 28411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:14.776054 28411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:14.780947 28419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:14.780928 28416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:14.780951 28417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:14.781347 28411 server_base.cc:1034] running on GCE node
I20260618 23:30:14.781580 28411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:14.781867 28411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:14.783028 28411 hybrid_clock.cc:648] HybridClock initialized: now 1781825414783018 us; error 47 us; skew 500 ppm
Jun 18 23:30:14 dist-test-slave-47qt krb5kdc[28398](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825414, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:14.785605 28411 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 23:30:14.786041 28411 webserver.cc:460] Webserver started at http://127.0.0.1:45569/ using document root <none> and password file <none>
I20260618 23:30:14.786257 28411 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:14.786299 28411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:14.786507 28411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:14.787536 28411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "33d64b6c8a8d4bb29d46e7c28dff9521"
format_stamp: "Formatted at 2026-06-18 23:30:14 on dist-test-slave-47qt"
server_key: "88324cc053106d5492dd9c84c64b82f3"
server_key_iv: "ad345354d666dfd8aef69c2f1580509e"
server_key_version: "encryptionkey@0"
I20260618 23:30:14.787849 28411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "33d64b6c8a8d4bb29d46e7c28dff9521"
format_stamp: "Formatted at 2026-06-18 23:30:14 on dist-test-slave-47qt"
server_key: "88324cc053106d5492dd9c84c64b82f3"
server_key_iv: "ad345354d666dfd8aef69c2f1580509e"
server_key_version: "encryptionkey@0"
I20260618 23:30:14.789727 28411 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260618 23:30:14.790709 28426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:14.790977 28411 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:30:14.791079 28411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "33d64b6c8a8d4bb29d46e7c28dff9521"
format_stamp: "Formatted at 2026-06-18 23:30:14 on dist-test-slave-47qt"
server_key: "88324cc053106d5492dd9c84c64b82f3"
server_key_iv: "ad345354d666dfd8aef69c2f1580509e"
server_key_version: "encryptionkey@0"
I20260618 23:30:14.791179 28411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:14.802693 28411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:14.805079 28411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:14.805251 28411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:14.811126 28411 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36943
I20260618 23:30:14.811177 28478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36943 every 8 connection(s)
I20260618 23:30:14.811530 28411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:30:14.812496 28479 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:14.814800 28479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33d64b6c8a8d4bb29d46e7c28dff9521: Bootstrap starting.
I20260618 23:30:14.815091 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 28411
I20260618 23:30:14.815223 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:30:14.815644 8350 external_mini_cluster.cc:1428] Setting key a21866ea793a477eb8f7b6aeec61a8d9
I20260618 23:30:14.815794 28479 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33d64b6c8a8d4bb29d46e7c28dff9521: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:14.816170 28479 log.cc:824] T 00000000000000000000000000000000 P 33d64b6c8a8d4bb29d46e7c28dff9521: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:14.816924 28479 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33d64b6c8a8d4bb29d46e7c28dff9521: No bootstrap required, opened a new log
I20260618 23:30:14.819258 28479 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33d64b6c8a8d4bb29d46e7c28dff9521 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d64b6c8a8d4bb29d46e7c28dff9521" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36943 } }
I20260618 23:30:14.819432 28479 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33d64b6c8a8d4bb29d46e7c28dff9521 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:14.819465 28479 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33d64b6c8a8d4bb29d46e7c28dff9521 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33d64b6c8a8d4bb29d46e7c28dff9521, State: Initialized, Role: FOLLOWER
I20260618 23:30:14.819579 28479 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33d64b6c8a8d4bb29d46e7c28dff9521 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d64b6c8a8d4bb29d46e7c28dff9521" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36943 } }
I20260618 23:30:14.819645 28479 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33d64b6c8a8d4bb29d46e7c28dff9521 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:30:14.819671 28479 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33d64b6c8a8d4bb29d46e7c28dff9521 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:30:14.819715 28479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33d64b6c8a8d4bb29d46e7c28dff9521 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:14.820292 28479 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33d64b6c8a8d4bb29d46e7c28dff9521 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d64b6c8a8d4bb29d46e7c28dff9521" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36943 } }
I20260618 23:30:14.820433 28479 leader_election.cc:302] T 00000000000000000000000000000000 P 33d64b6c8a8d4bb29d46e7c28dff9521 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33d64b6c8a8d4bb29d46e7c28dff9521; no voters:
I20260618 23:30:14.820616 28479 leader_election.cc:288] T 00000000000000000000000000000000 P 33d64b6c8a8d4bb29d46e7c28dff9521 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:30:14.820667 28484 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33d64b6c8a8d4bb29d46e7c28dff9521 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:30:14.820822 28484 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33d64b6c8a8d4bb29d46e7c28dff9521 [term 1 LEADER]: Becoming Leader. State: Replica: 33d64b6c8a8d4bb29d46e7c28dff9521, State: Running, Role: LEADER
I20260618 23:30:14.820904 28479 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33d64b6c8a8d4bb29d46e7c28dff9521 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:30:14.821007 28484 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33d64b6c8a8d4bb29d46e7c28dff9521 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d64b6c8a8d4bb29d46e7c28dff9521" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36943 } }
I20260618 23:30:14.821376 28485 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33d64b6c8a8d4bb29d46e7c28dff9521 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33d64b6c8a8d4bb29d46e7c28dff9521. Latest consensus state: current_term: 1 leader_uuid: "33d64b6c8a8d4bb29d46e7c28dff9521" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d64b6c8a8d4bb29d46e7c28dff9521" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36943 } } }
I20260618 23:30:14.821542 28485 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33d64b6c8a8d4bb29d46e7c28dff9521 [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:14.821815 28493 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:30:14.822325 28493 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:30:14.822372 28486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33d64b6c8a8d4bb29d46e7c28dff9521 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33d64b6c8a8d4bb29d46e7c28dff9521" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33d64b6c8a8d4bb29d46e7c28dff9521" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36943 } } }
I20260618 23:30:14.822551 28486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33d64b6c8a8d4bb29d46e7c28dff9521 [sys.catalog]: This master's current role is: LEADER
Jun 18 23:30:14 dist-test-slave-47qt krb5kdc[28398](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825414, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260618 23:30:14.824724 28493 catalog_manager.cc:1347] Generated new cluster ID: 4957d6c3f43e416998735047403dfb49
I20260618 23:30:14.824775 28493 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:30:14.827100 28482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:14.816217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38987 (local address 127.0.0.1:36943)
0618 23:30:14.816423 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:14.816437 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:14.816461 (+ 24us) server_negotiation.cc:407] Connection header received
0618 23:30:14.816902 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:14.816908 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:14.816953 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:14.817110 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:14.817930 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:14.818817 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:14.819873 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:14.820274 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:14.820368 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:14.823353 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:14.823357 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:14.823357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:14.823369 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:14.825501 (+ 2132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:14.825910 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:14.825913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:14.825913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:14.826010 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:14.826220 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:14.826220 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:14.826221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:14.826503 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:14.826642 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:14.826763 (+ 121us) server_negotiation.cc:299] Negotiation successful
0618 23:30:14.826897 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":89,"threads_started":1}
I20260618 23:30:14.827947 8350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:30:14.833262 28493 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:30:14.833763 28493 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:30:14.837217 28482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:14.829430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41833 (local address 127.0.0.1:36943)
0618 23:30:14.829476 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:14.829479 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:14.829598 (+ 119us) server_negotiation.cc:407] Connection header received
0618 23:30:14.829654 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:14.829655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:14.829667 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:14.829731 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:14.830362 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:14.831136 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:14.832132 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:14.832453 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:14.832521 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:14.833922 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:14.833924 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:14.833924 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:14.833931 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:14.835958 (+ 2027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:14.836348 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:14.836350 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:14.836350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:14.836405 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:14.836597 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:14.836598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:14.836598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:14.836714 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:14.836783 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:14.836995 (+ 212us) server_negotiation.cc:299] Negotiation successful
0618 23:30:14.837094 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260618 23:30:14.837993 28493 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33d64b6c8a8d4bb29d46e7c28dff9521: Generated new TSK 0
I20260618 23:30:14.838162 28493 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:30:14.840831 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 28411
2026-06-18T23:30:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1332 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:30:14 dist-test-slave-47qt krb5kdc[28510](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:30:14 dist-test-slave-47qt krb5kdc[28510](info): set up 2 sockets
Jun 18 23:30:14 dist-test-slave-47qt krb5kdc[28510](info): commencing operation
krb5kdc: starting...
W20260618 23:30:15.905172 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:30:15 dist-test-slave-47qt krb5kdc[28510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825415, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:30:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:30:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 23:30:16.033274 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42581
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:30:16.117444 28523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:16.117659 28523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:16.117686 28523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:16.118894 28523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:30:16.118948 28523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:16.118963 28523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:16.118976 28523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:30:16.118989 28523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:30:16.120481 28523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28523
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:16.120741 28523 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:16.120985 28523 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:16.125874 28528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:16.126031 28531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:16.126011 28529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:16.126444 28523 server_base.cc:1034] running on GCE node
I20260618 23:30:16.126859 28523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:16.127140 28523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:16.128373 28523 hybrid_clock.cc:648] HybridClock initialized: now 1781825416128349 us; error 35 us; skew 500 ppm
Jun 18 23:30:16 dist-test-slave-47qt krb5kdc[28510](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825416, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:16.139230 28523 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 23:30:16.139789 28523 webserver.cc:460] Webserver started at http://127.0.0.1:34953/ using document root <none> and password file <none>
I20260618 23:30:16.139993 28523 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:16.140038 28523 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:16.140127 28523 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:16.141762 28523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "9bf6418928554daa907c4cc23361406d"
format_stamp: "Formatted at 2026-06-18 23:30:16 on dist-test-slave-47qt"
server_key: "57c76489b8f16b68af06e55f46436390"
server_key_iv: "e79874ea93f13207b5a30a47959d1f2f"
server_key_version: "encryptionkey@0"
I20260618 23:30:16.142444 28523 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "9bf6418928554daa907c4cc23361406d"
format_stamp: "Formatted at 2026-06-18 23:30:16 on dist-test-slave-47qt"
server_key: "57c76489b8f16b68af06e55f46436390"
server_key_iv: "e79874ea93f13207b5a30a47959d1f2f"
server_key_version: "encryptionkey@0"
I20260618 23:30:16.143975 28523 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:30:16.144842 28538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:16.145059 28523 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:30:16.145213 28523 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "9bf6418928554daa907c4cc23361406d"
format_stamp: "Formatted at 2026-06-18 23:30:16 on dist-test-slave-47qt"
server_key: "57c76489b8f16b68af06e55f46436390"
server_key_iv: "e79874ea93f13207b5a30a47959d1f2f"
server_key_version: "encryptionkey@0"
I20260618 23:30:16.145282 28523 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:16.157779 28523 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:16.159868 28523 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:16.159987 28523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:16.163800 28523 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42581
I20260618 23:30:16.163887 28590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42581 every 8 connection(s)
I20260618 23:30:16.164204 28523 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:30:16.165774 28591 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:16.168295 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 28523
I20260618 23:30:16.168395 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:30:16.168612 8350 external_mini_cluster.cc:1428] Setting key 7ded4ea392db4142852ccf756c6949ba
I20260618 23:30:16.169519 28591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bf6418928554daa907c4cc23361406d: Bootstrap starting.
I20260618 23:30:16.170485 28591 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bf6418928554daa907c4cc23361406d: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:16.170815 28591 log.cc:824] T 00000000000000000000000000000000 P 9bf6418928554daa907c4cc23361406d: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:16.171620 28591 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bf6418928554daa907c4cc23361406d: No bootstrap required, opened a new log
I20260618 23:30:16.173928 28591 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bf6418928554daa907c4cc23361406d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf6418928554daa907c4cc23361406d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42581 } }
I20260618 23:30:16.174125 28591 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bf6418928554daa907c4cc23361406d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:16.174165 28591 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bf6418928554daa907c4cc23361406d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bf6418928554daa907c4cc23361406d, State: Initialized, Role: FOLLOWER
I20260618 23:30:16.174288 28591 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bf6418928554daa907c4cc23361406d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf6418928554daa907c4cc23361406d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42581 } }
I20260618 23:30:16.174374 28591 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bf6418928554daa907c4cc23361406d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:30:16.174407 28591 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bf6418928554daa907c4cc23361406d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:30:16.174474 28591 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bf6418928554daa907c4cc23361406d [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:16.175297 28594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:16.169253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46267 (local address 127.0.0.1:42581)
0618 23:30:16.169476 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:16.169489 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:16.169529 (+ 40us) server_negotiation.cc:407] Connection header received
0618 23:30:16.170427 (+ 898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:16.170439 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:16.170514 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:16.170704 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:16.170925 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:16.172253 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:16.173059 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:16.173763 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:16.173879 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:16.174626 (+ 747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:16.174631 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:16.174631 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:16.174830 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:16.174865 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:16.174879 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:16.174953 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:16.175010 (+ 57us) server_negotiation.cc:299] Negotiation successful
0618 23:30:16.175077 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":69,"threads_started":1}
I20260618 23:30:16.175319 28591 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bf6418928554daa907c4cc23361406d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf6418928554daa907c4cc23361406d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42581 } }
I20260618 23:30:16.175408 28591 leader_election.cc:302] T 00000000000000000000000000000000 P 9bf6418928554daa907c4cc23361406d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bf6418928554daa907c4cc23361406d; no voters:
I20260618 23:30:16.175581 28591 leader_election.cc:288] T 00000000000000000000000000000000 P 9bf6418928554daa907c4cc23361406d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:30:16.175633 28596 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bf6418928554daa907c4cc23361406d [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:30:16.175786 28596 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bf6418928554daa907c4cc23361406d [term 1 LEADER]: Becoming Leader. State: Replica: 9bf6418928554daa907c4cc23361406d, State: Running, Role: LEADER
I20260618 23:30:16.175928 28591 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bf6418928554daa907c4cc23361406d [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:30:16.175908 28596 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bf6418928554daa907c4cc23361406d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf6418928554daa907c4cc23361406d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42581 } }
I20260618 23:30:16.176419 8350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:30:16.176383 28597 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bf6418928554daa907c4cc23361406d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bf6418928554daa907c4cc23361406d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf6418928554daa907c4cc23361406d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42581 } } }
I20260618 23:30:16.176497 28597 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bf6418928554daa907c4cc23361406d [sys.catalog]: This master's current role is: LEADER
W20260618 23:30:16.177652 28611 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9bf6418928554daa907c4cc23361406d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 23:30:16.177739 28611 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:30:16.177855 28605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:30:16.176368 28598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bf6418928554daa907c4cc23361406d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bf6418928554daa907c4cc23361406d. Latest consensus state: current_term: 1 leader_uuid: "9bf6418928554daa907c4cc23361406d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bf6418928554daa907c4cc23361406d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42581 } } }
I20260618 23:30:16.178121 28598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bf6418928554daa907c4cc23361406d [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:16.178046 28605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:30:16.180220 28605 catalog_manager.cc:1347] Generated new cluster ID: 04bb83dd94de48fa8abc364e4511134b
I20260618 23:30:16.180296 28605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:30:16.181628 28594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:16.178014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51653 (local address 127.0.0.1:42581)
0618 23:30:16.178051 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:16.178053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:16.178137 (+ 84us) server_negotiation.cc:407] Connection header received
0618 23:30:16.178234 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:16.178235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:16.178246 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:16.178276 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:16.178542 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:16.179272 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:16.180225 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:16.180483 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:16.180556 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:16.181315 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:16.181316 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:16.181316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:16.181321 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:16.181335 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:16.181339 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:16.181382 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:16.181538 (+ 156us) server_negotiation.cc:299] Negotiation successful
0618 23:30:16.181558 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260618 23:30:16.190181 28605 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:30:16.190711 28605 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:30:16.193872 28605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bf6418928554daa907c4cc23361406d: Generated new TSK 0
I20260618 23:30:16.194067 28605 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:30:16.198586 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 28523
2026-06-18T23:30:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1358 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:30:16 dist-test-slave-47qt krb5kdc[28622](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:30:16 dist-test-slave-47qt krb5kdc[28622](info): set up 2 sockets
Jun 18 23:30:16 dist-test-slave-47qt krb5kdc[28622](info): commencing operation
krb5kdc: starting...
W20260618 23:30:17.265148 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:30:17 dist-test-slave-47qt krb5kdc[28622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825417, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:30:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:30:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 23:30:17.389709 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43075
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:30:17.470104 28635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:17.470338 28635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:17.470366 28635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:17.471495 28635 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:30:17.471534 28635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:17.471547 28635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:17.471560 28635 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:30:17.471572 28635 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:30:17.473194 28635 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28635
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:17.473405 28635 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:17.473664 28635 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:17.479285 28643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:17.479285 28640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:17.479338 28641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:17.480099 28635 server_base.cc:1034] running on GCE node
I20260618 23:30:17.480373 28635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:17.480710 28635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:17.481900 28635 hybrid_clock.cc:648] HybridClock initialized: now 1781825417481882 us; error 28 us; skew 500 ppm
Jun 18 23:30:17 dist-test-slave-47qt krb5kdc[28622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825417, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:17.484833 28635 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 23:30:17.485373 28635 webserver.cc:460] Webserver started at http://127.0.0.1:41359/ using document root <none> and password file <none>
I20260618 23:30:17.485603 28635 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:17.485672 28635 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:17.485831 28635 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:17.486898 28635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "ec4f7386eb18481688a7d568d05df357"
format_stamp: "Formatted at 2026-06-18 23:30:17 on dist-test-slave-47qt"
server_key: "f62142a52517d9c18319f4248658e4bd"
server_key_iv: "fdec7af2c1138ec1b99336b977c45103"
server_key_version: "encryptionkey@0"
I20260618 23:30:17.487278 28635 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "ec4f7386eb18481688a7d568d05df357"
format_stamp: "Formatted at 2026-06-18 23:30:17 on dist-test-slave-47qt"
server_key: "f62142a52517d9c18319f4248658e4bd"
server_key_iv: "fdec7af2c1138ec1b99336b977c45103"
server_key_version: "encryptionkey@0"
I20260618 23:30:17.488584 28635 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:30:17.489374 28650 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:17.489566 28635 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:30:17.489663 28635 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "ec4f7386eb18481688a7d568d05df357"
format_stamp: "Formatted at 2026-06-18 23:30:17 on dist-test-slave-47qt"
server_key: "f62142a52517d9c18319f4248658e4bd"
server_key_iv: "fdec7af2c1138ec1b99336b977c45103"
server_key_version: "encryptionkey@0"
I20260618 23:30:17.489745 28635 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:17.493180 28635 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:17.495343 28635 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:17.495528 28635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:17.499868 28635 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43075
I20260618 23:30:17.499913 28702 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43075 every 8 connection(s)
I20260618 23:30:17.500245 28635 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:30:17.501291 28703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:17.503671 28703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec4f7386eb18481688a7d568d05df357: Bootstrap starting.
I20260618 23:30:17.504289 28703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec4f7386eb18481688a7d568d05df357: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:17.504577 28703 log.cc:824] T 00000000000000000000000000000000 P ec4f7386eb18481688a7d568d05df357: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:17.504942 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 28635
I20260618 23:30:17.505069 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:30:17.505271 8350 external_mini_cluster.cc:1428] Setting key dc0b688f0f3df3eba933de0eac72ce97
I20260618 23:30:17.505429 28703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec4f7386eb18481688a7d568d05df357: No bootstrap required, opened a new log
I20260618 23:30:17.507964 28707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:17.505957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35171 (local address 127.0.0.1:43075)
0618 23:30:17.506157 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:17.506174 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:17.506198 (+ 24us) server_negotiation.cc:407] Connection header received
0618 23:30:17.506742 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:17.506749 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:17.506775 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:17.506887 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:17.507531 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:17.507534 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:17.507534 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:17.507538 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:17.507573 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:17.507583 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:17.507656 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:17.507719 (+ 63us) server_negotiation.cc:299] Negotiation successful
0618 23:30:17.507773 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":98,"threads_started":1}
I20260618 23:30:17.508380 28703 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec4f7386eb18481688a7d568d05df357 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec4f7386eb18481688a7d568d05df357" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43075 } }
I20260618 23:30:17.508571 28703 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec4f7386eb18481688a7d568d05df357 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:17.508610 28703 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec4f7386eb18481688a7d568d05df357 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec4f7386eb18481688a7d568d05df357, State: Initialized, Role: FOLLOWER
I20260618 23:30:17.508737 28703 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec4f7386eb18481688a7d568d05df357 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec4f7386eb18481688a7d568d05df357" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43075 } }
I20260618 23:30:17.508818 28703 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec4f7386eb18481688a7d568d05df357 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:30:17.508838 28703 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec4f7386eb18481688a7d568d05df357 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:30:17.508865 28703 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec4f7386eb18481688a7d568d05df357 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:17.509549 28703 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec4f7386eb18481688a7d568d05df357 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec4f7386eb18481688a7d568d05df357" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43075 } }
I20260618 23:30:17.509685 28703 leader_election.cc:302] T 00000000000000000000000000000000 P ec4f7386eb18481688a7d568d05df357 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec4f7386eb18481688a7d568d05df357; no voters:
I20260618 23:30:17.509861 28703 leader_election.cc:288] T 00000000000000000000000000000000 P ec4f7386eb18481688a7d568d05df357 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:30:17.509955 28708 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec4f7386eb18481688a7d568d05df357 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:30:17.510072 8350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:30:17.510138 28703 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec4f7386eb18481688a7d568d05df357 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:30:17.510210 28708 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec4f7386eb18481688a7d568d05df357 [term 1 LEADER]: Becoming Leader. State: Replica: ec4f7386eb18481688a7d568d05df357, State: Running, Role: LEADER
I20260618 23:30:17.510372 28708 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec4f7386eb18481688a7d568d05df357 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec4f7386eb18481688a7d568d05df357" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43075 } }
I20260618 23:30:17.511003 28710 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec4f7386eb18481688a7d568d05df357 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec4f7386eb18481688a7d568d05df357" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec4f7386eb18481688a7d568d05df357" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43075 } } }
I20260618 23:30:17.510982 28708 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec4f7386eb18481688a7d568d05df357 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec4f7386eb18481688a7d568d05df357. Latest consensus state: current_term: 1 leader_uuid: "ec4f7386eb18481688a7d568d05df357" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec4f7386eb18481688a7d568d05df357" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43075 } } }
I20260618 23:30:17.511101 28710 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec4f7386eb18481688a7d568d05df357 [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:17.511102 28708 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec4f7386eb18481688a7d568d05df357 [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:17.511698 28720 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:30:17.512264 28720 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:30:17.513909 28707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:17.511846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40497 (local address 127.0.0.1:43075)
0618 23:30:17.511881 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:17.511882 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:17.512874 (+ 992us) server_negotiation.cc:407] Connection header received
0618 23:30:17.512935 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:17.512936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:17.512963 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:17.513039 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:17.513683 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:17.513685 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:17.513685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:17.513689 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:17.513704 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:17.513713 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:17.513774 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:17.513831 (+ 57us) server_negotiation.cc:299] Negotiation successful
0618 23:30:17.513846 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260618 23:30:17.514272 28720 catalog_manager.cc:1347] Generated new cluster ID: 19ead582c2874e7d8bae23986e91c665
I20260618 23:30:17.514324 28720 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:30:17.520205 28720 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:30:17.520385 28720 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:30:17.533314 28720 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec4f7386eb18481688a7d568d05df357: Generated new TSK 0
I20260618 23:30:17.533573 28720 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:30:17.541296 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 28635
2026-06-18T23:30:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1343 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:30:17 dist-test-slave-47qt krb5kdc[28734](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:30:17 dist-test-slave-47qt krb5kdc[28734](info): set up 2 sockets
Jun 18 23:30:17 dist-test-slave-47qt krb5kdc[28734](info): commencing operation
krb5kdc: starting...
W20260618 23:30:18.625174 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:30:18 dist-test-slave-47qt krb5kdc[28734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:30:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:30:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 23:30:18.756033 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44523
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46443
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44523
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:30:18.842566 28747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:18.843093 28747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:18.843154 28747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:18.845523 28747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:30:18.845659 28747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:18.845686 28747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:18.845709 28747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:30:18.845742 28747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:30:18.847846 28747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46443
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:18.848484 28747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:18.848786 28747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:18.854003 28755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:18.854111 28753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:18.854111 28752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:18.854482 28747 server_base.cc:1034] running on GCE node
I20260618 23:30:18.854784 28747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:18.855100 28747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:18.856294 28747 hybrid_clock.cc:648] HybridClock initialized: now 1781825418856272 us; error 45 us; skew 500 ppm
Jun 18 23:30:18 dist-test-slave-47qt krb5kdc[28734](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825418, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:18.859229 28747 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 23:30:18.859864 28747 webserver.cc:460] Webserver started at http://127.0.0.1:43385/ using document root <none> and password file <none>
I20260618 23:30:18.860169 28747 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:18.860240 28747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:18.860397 28747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:18.863022 28747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "0df91b0a454b4c70ae8f5431582ad7df"
format_stamp: "Formatted at 2026-06-18 23:30:18 on dist-test-slave-47qt"
server_key: "c6af09fa30803e4425638d6771045ce4"
server_key_iv: "5c3b00e2d2652e6fe7f65e3a457babaa"
server_key_version: "encryptionkey@0"
I20260618 23:30:18.864040 28747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "0df91b0a454b4c70ae8f5431582ad7df"
format_stamp: "Formatted at 2026-06-18 23:30:18 on dist-test-slave-47qt"
server_key: "c6af09fa30803e4425638d6771045ce4"
server_key_iv: "5c3b00e2d2652e6fe7f65e3a457babaa"
server_key_version: "encryptionkey@0"
I20260618 23:30:18.866354 28747 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.005s
I20260618 23:30:18.867218 28762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:18.867394 28747 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:30:18.867513 28747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "0df91b0a454b4c70ae8f5431582ad7df"
format_stamp: "Formatted at 2026-06-18 23:30:18 on dist-test-slave-47qt"
server_key: "c6af09fa30803e4425638d6771045ce4"
server_key_iv: "5c3b00e2d2652e6fe7f65e3a457babaa"
server_key_version: "encryptionkey@0"
I20260618 23:30:18.867614 28747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:18.879549 28747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:18.882383 28747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:18.882552 28747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:18.887398 28747 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44523
I20260618 23:30:18.887430 28814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44523 every 8 connection(s)
I20260618 23:30:18.887835 28747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:30:18.888847 28815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:18.891156 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 28747
I20260618 23:30:18.891268 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:30:18.891374 28815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0df91b0a454b4c70ae8f5431582ad7df: Bootstrap starting.
I20260618 23:30:18.891490 8350 external_mini_cluster.cc:1428] Setting key ec8523d01aaa146e0f49a74d5b2e76ce
I20260618 23:30:18.892174 28815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0df91b0a454b4c70ae8f5431582ad7df: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:18.892510 28815 log.cc:824] T 00000000000000000000000000000000 P 0df91b0a454b4c70ae8f5431582ad7df: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:18.893321 28815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0df91b0a454b4c70ae8f5431582ad7df: No bootstrap required, opened a new log
I20260618 23:30:18.895478 28815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0df91b0a454b4c70ae8f5431582ad7df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df91b0a454b4c70ae8f5431582ad7df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44523 } }
I20260618 23:30:18.895663 28815 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0df91b0a454b4c70ae8f5431582ad7df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:18.895704 28815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0df91b0a454b4c70ae8f5431582ad7df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0df91b0a454b4c70ae8f5431582ad7df, State: Initialized, Role: FOLLOWER
I20260618 23:30:18.895884 28815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0df91b0a454b4c70ae8f5431582ad7df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df91b0a454b4c70ae8f5431582ad7df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44523 } }
I20260618 23:30:18.896008 28820 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0df91b0a454b4c70ae8f5431582ad7df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260618 23:30:18.896153 28820 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0df91b0a454b4c70ae8f5431582ad7df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df91b0a454b4c70ae8f5431582ad7df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44523 } }
I20260618 23:30:18.895979 28815 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0df91b0a454b4c70ae8f5431582ad7df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:30:18.896279 28820 leader_election.cc:302] T 00000000000000000000000000000000 P 0df91b0a454b4c70ae8f5431582ad7df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0df91b0a454b4c70ae8f5431582ad7df; no voters:
I20260618 23:30:18.896325 28815 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0df91b0a454b4c70ae8f5431582ad7df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:30:18.896358 28815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0df91b0a454b4c70ae8f5431582ad7df [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:18.896458 28820 leader_election.cc:288] T 00000000000000000000000000000000 P 0df91b0a454b4c70ae8f5431582ad7df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260618 23:30:18.897109 28815 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0df91b0a454b4c70ae8f5431582ad7df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df91b0a454b4c70ae8f5431582ad7df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44523 } }
I20260618 23:30:18.897243 28815 leader_election.cc:302] T 00000000000000000000000000000000 P 0df91b0a454b4c70ae8f5431582ad7df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0df91b0a454b4c70ae8f5431582ad7df; no voters:
I20260618 23:30:18.897269 28821 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 0df91b0a454b4c70ae8f5431582ad7df [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260618 23:30:18.897320 28815 leader_election.cc:288] T 00000000000000000000000000000000 P 0df91b0a454b4c70ae8f5431582ad7df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:30:18.897514 28820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0df91b0a454b4c70ae8f5431582ad7df [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:30:18.897511 28821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0df91b0a454b4c70ae8f5431582ad7df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df91b0a454b4c70ae8f5431582ad7df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44523 } } }
I20260618 23:30:18.897588 28820 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0df91b0a454b4c70ae8f5431582ad7df [term 1 LEADER]: Becoming Leader. State: Replica: 0df91b0a454b4c70ae8f5431582ad7df, State: Running, Role: LEADER
I20260618 23:30:18.897594 28821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0df91b0a454b4c70ae8f5431582ad7df [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:30:18.897666 28815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0df91b0a454b4c70ae8f5431582ad7df [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:30:18.897666 28820 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0df91b0a454b4c70ae8f5431582ad7df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df91b0a454b4c70ae8f5431582ad7df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44523 } }
I20260618 23:30:18.898124 28821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0df91b0a454b4c70ae8f5431582ad7df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0df91b0a454b4c70ae8f5431582ad7df. Latest consensus state: current_term: 1 leader_uuid: "0df91b0a454b4c70ae8f5431582ad7df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df91b0a454b4c70ae8f5431582ad7df" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44523 } } }
I20260618 23:30:18.898242 28821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0df91b0a454b4c70ae8f5431582ad7df [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:18.898856 28829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 18 23:30:18 dist-test-slave-47qt krb5kdc[28734](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260618 23:30:18.899372 28829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:30:18.900885 28829 catalog_manager.cc:1347] Generated new cluster ID: 968b55d7c32c483f8c6e430e558a0669
I20260618 23:30:18.900940 28829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:30:18.904163 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:18.892112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:56379 (local address 127.0.0.1:44523)
0618 23:30:18.892296 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:18.892304 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:18.892329 (+ 25us) server_negotiation.cc:407] Connection header received
0618 23:30:18.892839 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:18.892846 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:18.892895 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:18.893013 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:18.893837 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:18.894727 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:18.895841 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:18.896451 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:18.896821 (+ 370us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:18.899670 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:18.899674 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:18.899674 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:18.899684 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:18.902184 (+ 2500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:18.902646 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:18.902649 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:18.902650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:18.902766 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:18.902997 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:18.902998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:18.902999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:18.903304 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:18.903463 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:18.903580 (+ 117us) server_negotiation.cc:299] Negotiation successful
0618 23:30:18.903900 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":96,"threads_started":1}
I20260618 23:30:18.905360 8350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:30:18.919479 28829 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:30:18.920107 28829 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:30:18.921777 28818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:18.909517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:37857 (local address 127.0.0.1:44523)
0618 23:30:18.909583 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:18.909585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:18.913495 (+ 3910us) server_negotiation.cc:407] Connection header received
0618 23:30:18.913696 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:18.913700 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:18.913733 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:18.913841 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:18.914797 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:18.915574 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:18.916634 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:18.916961 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:18.917024 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:18.918680 (+ 1656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:18.918682 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:18.918682 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:18.918691 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:18.920446 (+ 1755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:18.920899 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:18.920900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:18.920901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:18.920966 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:18.921213 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:18.921214 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:18.921214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:18.921349 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:18.921458 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:18.921606 (+ 148us) server_negotiation.cc:299] Negotiation successful
0618 23:30:18.921674 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260618 23:30:18.924053 28829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0df91b0a454b4c70ae8f5431582ad7df: Generated new TSK 0
I20260618 23:30:18.924252 28829 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:30:18.926601 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 28747
W20260618 23:30:18.931393 28732 connection.cc:535] client connection to 127.0.0.1:44523 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44523): Connection reset by peer (error 104)
2026-06-18T23:30:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:30:18 dist-test-slave-47qt krb5kdc[28846](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:30:18 dist-test-slave-47qt krb5kdc[28846](info): set up 2 sockets
Jun 18 23:30:18 dist-test-slave-47qt krb5kdc[28846](info): commencing operation
krb5kdc: starting...
W20260618 23:30:19.989583 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:30:20 dist-test-slave-47qt krb5kdc[28846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825420, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:30:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:30:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 23:30:20.116422 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38561
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:30:20.201581 28859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:20.201820 28859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:20.201858 28859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:20.203128 28859 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:30:20.203223 28859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:20.203263 28859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:20.203332 28859 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:30:20.203357 28859 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:30:20.205076 28859 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38561
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:20.205377 28859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:20.205611 28859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:20.211031 28865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:20.211129 28859 server_base.cc:1034] running on GCE node
W20260618 23:30:20.211087 28864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:20.211024 28867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:20.211540 28859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:20.211858 28859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:20.213050 28859 hybrid_clock.cc:648] HybridClock initialized: now 1781825420213007 us; error 59 us; skew 500 ppm
Jun 18 23:30:20 dist-test-slave-47qt krb5kdc[28846](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825420, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:20.215725 28859 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 23:30:20.216208 28859 webserver.cc:460] Webserver started at http://127.0.0.1:37613/ using document root <none> and password file <none>
I20260618 23:30:20.216424 28859 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:20.216466 28859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:20.216554 28859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:20.217645 28859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "dd1be6b49fff423f9ab37d1b5087c5a5"
format_stamp: "Formatted at 2026-06-18 23:30:20 on dist-test-slave-47qt"
server_key: "5c03b35d7f6360da4a7ae34c0a4aa614"
server_key_iv: "ec872fbfe35755e34b28fb0bdaa69a39"
server_key_version: "encryptionkey@0"
I20260618 23:30:20.217957 28859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "dd1be6b49fff423f9ab37d1b5087c5a5"
format_stamp: "Formatted at 2026-06-18 23:30:20 on dist-test-slave-47qt"
server_key: "5c03b35d7f6360da4a7ae34c0a4aa614"
server_key_iv: "ec872fbfe35755e34b28fb0bdaa69a39"
server_key_version: "encryptionkey@0"
I20260618 23:30:20.219240 28859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:30:20.220026 28874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:20.220238 28859 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:30:20.220338 28859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "dd1be6b49fff423f9ab37d1b5087c5a5"
format_stamp: "Formatted at 2026-06-18 23:30:20 on dist-test-slave-47qt"
server_key: "5c03b35d7f6360da4a7ae34c0a4aa614"
server_key_iv: "ec872fbfe35755e34b28fb0bdaa69a39"
server_key_version: "encryptionkey@0"
I20260618 23:30:20.220432 28859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:20.238246 28859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:20.240609 28859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:20.240918 28859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:20.245067 28859 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38561
I20260618 23:30:20.245174 28926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38561 every 8 connection(s)
I20260618 23:30:20.245500 28859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:30:20.246551 28927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:20.249063 28927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd1be6b49fff423f9ab37d1b5087c5a5: Bootstrap starting.
I20260618 23:30:20.249703 28927 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd1be6b49fff423f9ab37d1b5087c5a5: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:20.250001 28927 log.cc:824] T 00000000000000000000000000000000 P dd1be6b49fff423f9ab37d1b5087c5a5: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:20.251168 28927 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd1be6b49fff423f9ab37d1b5087c5a5: No bootstrap required, opened a new log
I20260618 23:30:20.252408 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 28859
I20260618 23:30:20.252539 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:30:20.252714 8350 external_mini_cluster.cc:1428] Setting key 7629997755494af06050c96620608c3e
I20260618 23:30:20.253695 28927 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd1be6b49fff423f9ab37d1b5087c5a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1be6b49fff423f9ab37d1b5087c5a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38561 } }
I20260618 23:30:20.253872 28927 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd1be6b49fff423f9ab37d1b5087c5a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:20.253934 28927 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd1be6b49fff423f9ab37d1b5087c5a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd1be6b49fff423f9ab37d1b5087c5a5, State: Initialized, Role: FOLLOWER
I20260618 23:30:20.254115 28927 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd1be6b49fff423f9ab37d1b5087c5a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1be6b49fff423f9ab37d1b5087c5a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38561 } }
I20260618 23:30:20.254205 28927 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd1be6b49fff423f9ab37d1b5087c5a5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:30:20.254251 28927 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd1be6b49fff423f9ab37d1b5087c5a5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:30:20.254307 28927 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd1be6b49fff423f9ab37d1b5087c5a5 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:20.254895 28927 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd1be6b49fff423f9ab37d1b5087c5a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1be6b49fff423f9ab37d1b5087c5a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38561 } }
I20260618 23:30:20.255013 28927 leader_election.cc:302] T 00000000000000000000000000000000 P dd1be6b49fff423f9ab37d1b5087c5a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd1be6b49fff423f9ab37d1b5087c5a5; no voters:
I20260618 23:30:20.255193 28927 leader_election.cc:288] T 00000000000000000000000000000000 P dd1be6b49fff423f9ab37d1b5087c5a5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:30:20.255239 28932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd1be6b49fff423f9ab37d1b5087c5a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:30:20.255371 28932 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd1be6b49fff423f9ab37d1b5087c5a5 [term 1 LEADER]: Becoming Leader. State: Replica: dd1be6b49fff423f9ab37d1b5087c5a5, State: Running, Role: LEADER
I20260618 23:30:20.255455 28927 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd1be6b49fff423f9ab37d1b5087c5a5 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:30:20.255502 28932 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd1be6b49fff423f9ab37d1b5087c5a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1be6b49fff423f9ab37d1b5087c5a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38561 } }
I20260618 23:30:20.255897 28932 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd1be6b49fff423f9ab37d1b5087c5a5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd1be6b49fff423f9ab37d1b5087c5a5. Latest consensus state: current_term: 1 leader_uuid: "dd1be6b49fff423f9ab37d1b5087c5a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1be6b49fff423f9ab37d1b5087c5a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38561 } } }
I20260618 23:30:20.256101 28932 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd1be6b49fff423f9ab37d1b5087c5a5 [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:20.256042 28934 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd1be6b49fff423f9ab37d1b5087c5a5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd1be6b49fff423f9ab37d1b5087c5a5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd1be6b49fff423f9ab37d1b5087c5a5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38561 } } }
I20260618 23:30:20.256318 28934 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd1be6b49fff423f9ab37d1b5087c5a5 [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:20.256517 28941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:30:20.257149 28941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:30:20.258735 28941 catalog_manager.cc:1347] Generated new cluster ID: 2c313e5365bd4bf89cb1b6f77ee418c4
I20260618 23:30:20.258785 28941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:30:20.259109 28931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:20.253329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:44783 (local address 127.0.0.1:38561)
0618 23:30:20.253528 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:20.253536 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:20.253565 (+ 29us) server_negotiation.cc:407] Connection header received
0618 23:30:20.253664 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:20.253669 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:20.253717 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:20.253868 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:20.254080 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:20.255519 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:20.256641 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:20.257473 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:20.257650 (+ 177us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:20.258398 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:20.258401 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:20.258401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:20.258586 (+ 185us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:20.258638 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:20.258654 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:20.258792 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:20.258879 (+ 87us) server_negotiation.cc:299] Negotiation successful
0618 23:30:20.258907 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":88,"threads_started":1}
I20260618 23:30:20.262066 8350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:30:20.267060 28931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:20.263627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:57075 (local address 127.0.0.1:38561)
0618 23:30:20.263667 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:20.263669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:20.263738 (+ 69us) server_negotiation.cc:407] Connection header received
0618 23:30:20.263816 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:20.263816 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:20.263837 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:20.263887 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:20.264017 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:20.264700 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:20.265729 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:20.266034 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:20.266096 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:20.266804 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:20.266805 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:20.266805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:20.266810 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:20.266823 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:20.266827 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:20.266878 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:20.266974 (+ 96us) server_negotiation.cc:299] Negotiation successful
0618 23:30:20.266985 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260618 23:30:20.272161 28941 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:30:20.272691 28941 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:30:20.278541 28941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd1be6b49fff423f9ab37d1b5087c5a5: Generated new TSK 0
I20260618 23:30:20.278718 28941 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:30:20.287166 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 28859
W20260618 23:30:20.291426 28844 connection.cc:535] client connection to 127.0.0.1:38561 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38561): Connection reset by peer (error 104)
2026-06-18T23:30:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1360 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:30:20 dist-test-slave-47qt krb5kdc[28958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:30:20 dist-test-slave-47qt krb5kdc[28958](info): set up 2 sockets
Jun 18 23:30:20 dist-test-slave-47qt krb5kdc[28958](info): commencing operation
krb5kdc: starting...
W20260618 23:30:21.369199 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.063s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:30:21 dist-test-slave-47qt krb5kdc[28958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825421, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:30:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:30:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 23:30:21.490779 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46283
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:30:21.574574 28971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:21.574772 28971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:21.574796 28971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:21.575958 28971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:30:21.576004 28971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:21.576018 28971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:21.576031 28971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:30:21.576045 28971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:30:21.577561 28971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:21.577806 28971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:21.578060 28971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:21.583197 28979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:21.583267 28977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:21.583199 28976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:21.584373 28971 server_base.cc:1034] running on GCE node
I20260618 23:30:21.584625 28971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:21.584937 28971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:21.586108 28971 hybrid_clock.cc:648] HybridClock initialized: now 1781825421586098 us; error 39 us; skew 500 ppm
Jun 18 23:30:21 dist-test-slave-47qt krb5kdc[28958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825421, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:21.595642 28971 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 23:30:21.596222 28971 webserver.cc:460] Webserver started at http://127.0.0.1:36939/ using document root <none> and password file <none>
I20260618 23:30:21.596465 28971 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:21.596534 28971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:21.596697 28971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:21.597862 28971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "0162018d6abd4bca839c8cfc497a3b8d"
format_stamp: "Formatted at 2026-06-18 23:30:21 on dist-test-slave-47qt"
server_key: "54bbaaada6613e0546c58187d16d89a3"
server_key_iv: "40e2290b37ac538ccf426cf02b270635"
server_key_version: "encryptionkey@0"
I20260618 23:30:21.598228 28971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "0162018d6abd4bca839c8cfc497a3b8d"
format_stamp: "Formatted at 2026-06-18 23:30:21 on dist-test-slave-47qt"
server_key: "54bbaaada6613e0546c58187d16d89a3"
server_key_iv: "40e2290b37ac538ccf426cf02b270635"
server_key_version: "encryptionkey@0"
I20260618 23:30:21.599896 28971 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:30:21.601100 28986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:21.601312 28971 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:30:21.601429 28971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "0162018d6abd4bca839c8cfc497a3b8d"
format_stamp: "Formatted at 2026-06-18 23:30:21 on dist-test-slave-47qt"
server_key: "54bbaaada6613e0546c58187d16d89a3"
server_key_iv: "40e2290b37ac538ccf426cf02b270635"
server_key_version: "encryptionkey@0"
I20260618 23:30:21.601524 28971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:21.606272 28971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:21.609719 28971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:21.610028 28971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:21.614296 28971 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46283
I20260618 23:30:21.614354 29038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46283 every 8 connection(s)
I20260618 23:30:21.614686 28971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:30:21.615767 29039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:21.616746 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 28971
I20260618 23:30:21.616878 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:30:21.617066 8350 external_mini_cluster.cc:1428] Setting key 7e9180878c4b142f6cefabadfb47a389
I20260618 23:30:21.618510 29039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0162018d6abd4bca839c8cfc497a3b8d: Bootstrap starting.
I20260618 23:30:21.619582 29039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0162018d6abd4bca839c8cfc497a3b8d: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:21.619751 29042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:21.617661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:59307 (local address 127.0.0.1:46283)
0618 23:30:21.617835 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:21.617852 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:21.617886 (+ 34us) server_negotiation.cc:407] Connection header received
0618 23:30:21.618302 (+ 416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:21.618307 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:21.618330 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:21.618444 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:21.619192 (+ 748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:21.619196 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:21.619196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:21.619200 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:21.619240 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:21.619251 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:21.619333 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:21.619401 (+ 68us) server_negotiation.cc:299] Negotiation successful
0618 23:30:21.619474 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
I20260618 23:30:21.619940 29039 log.cc:824] T 00000000000000000000000000000000 P 0162018d6abd4bca839c8cfc497a3b8d: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:21.620922 29039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0162018d6abd4bca839c8cfc497a3b8d: No bootstrap required, opened a new log
I20260618 23:30:21.621330 8350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:30:21.624284 29039 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0162018d6abd4bca839c8cfc497a3b8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0162018d6abd4bca839c8cfc497a3b8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46283 } }
I20260618 23:30:21.624670 29039 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0162018d6abd4bca839c8cfc497a3b8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:21.624754 29039 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0162018d6abd4bca839c8cfc497a3b8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0162018d6abd4bca839c8cfc497a3b8d, State: Initialized, Role: FOLLOWER
I20260618 23:30:21.624945 29039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0162018d6abd4bca839c8cfc497a3b8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0162018d6abd4bca839c8cfc497a3b8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46283 } }
I20260618 23:30:21.625084 29039 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0162018d6abd4bca839c8cfc497a3b8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:30:21.625177 29039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0162018d6abd4bca839c8cfc497a3b8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:30:21.625188 29042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:21.623353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:53481 (local address 127.0.0.1:46283)
0618 23:30:21.623518 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:21.623522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:21.623806 (+ 284us) server_negotiation.cc:407] Connection header received
0618 23:30:21.623867 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:21.623868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:21.623873 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:21.623944 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:21.624771 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:21.624772 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:21.624773 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:21.624777 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:21.624792 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:21.624796 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:21.624859 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:21.625037 (+ 178us) server_negotiation.cc:299] Negotiation successful
0618 23:30:21.625055 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260618 23:30:21.625234 29039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0162018d6abd4bca839c8cfc497a3b8d [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:21.625918 29039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0162018d6abd4bca839c8cfc497a3b8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0162018d6abd4bca839c8cfc497a3b8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46283 } }
I20260618 23:30:21.626044 29039 leader_election.cc:302] T 00000000000000000000000000000000 P 0162018d6abd4bca839c8cfc497a3b8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0162018d6abd4bca839c8cfc497a3b8d; no voters:
I20260618 23:30:21.626220 29039 leader_election.cc:288] T 00000000000000000000000000000000 P 0162018d6abd4bca839c8cfc497a3b8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:30:21.626255 29049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0162018d6abd4bca839c8cfc497a3b8d [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:30:21.626442 29049 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0162018d6abd4bca839c8cfc497a3b8d [term 1 LEADER]: Becoming Leader. State: Replica: 0162018d6abd4bca839c8cfc497a3b8d, State: Running, Role: LEADER
I20260618 23:30:21.626490 29039 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0162018d6abd4bca839c8cfc497a3b8d [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:30:21.626538 29049 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0162018d6abd4bca839c8cfc497a3b8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0162018d6abd4bca839c8cfc497a3b8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46283 } }
I20260618 23:30:21.626936 29050 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0162018d6abd4bca839c8cfc497a3b8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0162018d6abd4bca839c8cfc497a3b8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0162018d6abd4bca839c8cfc497a3b8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46283 } } }
I20260618 23:30:21.627040 29050 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0162018d6abd4bca839c8cfc497a3b8d [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:21.627066 29051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0162018d6abd4bca839c8cfc497a3b8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0162018d6abd4bca839c8cfc497a3b8d. Latest consensus state: current_term: 1 leader_uuid: "0162018d6abd4bca839c8cfc497a3b8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0162018d6abd4bca839c8cfc497a3b8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46283 } } }
I20260618 23:30:21.627173 29051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0162018d6abd4bca839c8cfc497a3b8d [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:21.627246 29054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:30:21.627736 29054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:30:21.629179 29054 catalog_manager.cc:1347] Generated new cluster ID: f66cb447463d47718472d7b91c5d5498
I20260618 23:30:21.629236 29054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:30:21.637401 29054 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:30:21.637575 29054 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:30:21.643092 29054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0162018d6abd4bca839c8cfc497a3b8d: Generated new TSK 0
I20260618 23:30:21.643319 29054 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:30:21.650564 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 28971
2026-06-18T23:30:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1363 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:30:21 dist-test-slave-47qt krb5kdc[29070](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:30:21 dist-test-slave-47qt krb5kdc[29070](info): set up 2 sockets
Jun 18 23:30:21 dist-test-slave-47qt krb5kdc[29070](info): commencing operation
krb5kdc: starting...
W20260618 23:30:22.725605 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:30:22 dist-test-slave-47qt krb5kdc[29070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:30:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:30:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 23:30:22.856315 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37379
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:30:22.941254 29083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:22.941488 29083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:22.941525 29083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:22.942623 29083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:30:22.942691 29083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:22.942734 29083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:22.942760 29083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:30:22.942786 29083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:30:22.944199 29083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37379
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:22.944465 29083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:22.944681 29083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 23:30:22.949889 29083 server_base.cc:1034] running on GCE node
W20260618 23:30:22.949870 29091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:22.949857 29088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:22.949865 29089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:22.950348 29083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:22.950647 29083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:22.951797 29083 hybrid_clock.cc:648] HybridClock initialized: now 1781825422951767 us; error 43 us; skew 500 ppm
Jun 18 23:30:22 dist-test-slave-47qt krb5kdc[29070](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825422, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:22.954865 29083 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 23:30:22.955498 29083 webserver.cc:460] Webserver started at http://127.0.0.1:45691/ using document root <none> and password file <none>
I20260618 23:30:22.955777 29083 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:22.955829 29083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:22.956043 29083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:22.957885 29083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "a48f1da8eee34813ba07200ee0415c41"
format_stamp: "Formatted at 2026-06-18 23:30:22 on dist-test-slave-47qt"
server_key: "fd27ece87f47a66f8806a989ab617f69"
server_key_iv: "bc873be199158d3e08a9160ca4bc6a84"
server_key_version: "encryptionkey@0"
I20260618 23:30:22.958346 29083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "a48f1da8eee34813ba07200ee0415c41"
format_stamp: "Formatted at 2026-06-18 23:30:22 on dist-test-slave-47qt"
server_key: "fd27ece87f47a66f8806a989ab617f69"
server_key_iv: "bc873be199158d3e08a9160ca4bc6a84"
server_key_version: "encryptionkey@0"
I20260618 23:30:22.959836 29083 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260618 23:30:22.960947 29098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:22.961290 29083 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:30:22.961382 29083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "a48f1da8eee34813ba07200ee0415c41"
format_stamp: "Formatted at 2026-06-18 23:30:22 on dist-test-slave-47qt"
server_key: "fd27ece87f47a66f8806a989ab617f69"
server_key_iv: "bc873be199158d3e08a9160ca4bc6a84"
server_key_version: "encryptionkey@0"
I20260618 23:30:22.961486 29083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:22.975387 29083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:22.978224 29083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:22.978392 29083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:22.982905 29083 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37379
I20260618 23:30:22.982936 29150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37379 every 8 connection(s)
I20260618 23:30:22.983296 29083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:30:22.984272 29151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:22.986677 29151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a48f1da8eee34813ba07200ee0415c41: Bootstrap starting.
I20260618 23:30:22.987197 29151 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a48f1da8eee34813ba07200ee0415c41: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:22.987440 29151 log.cc:824] T 00000000000000000000000000000000 P a48f1da8eee34813ba07200ee0415c41: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:22.988173 29151 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a48f1da8eee34813ba07200ee0415c41: No bootstrap required, opened a new log
I20260618 23:30:22.989759 29151 raft_consensus.cc:348] T 00000000000000000000000000000000 P a48f1da8eee34813ba07200ee0415c41 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48f1da8eee34813ba07200ee0415c41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37379 } }
I20260618 23:30:22.989902 29151 raft_consensus.cc:374] T 00000000000000000000000000000000 P a48f1da8eee34813ba07200ee0415c41 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:22.989925 29151 raft_consensus.cc:729] T 00000000000000000000000000000000 P a48f1da8eee34813ba07200ee0415c41 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a48f1da8eee34813ba07200ee0415c41, State: Initialized, Role: FOLLOWER
I20260618 23:30:22.990025 29151 consensus_queue.cc:260] T 00000000000000000000000000000000 P a48f1da8eee34813ba07200ee0415c41 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48f1da8eee34813ba07200ee0415c41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37379 } }
I20260618 23:30:22.990127 29151 raft_consensus.cc:388] T 00000000000000000000000000000000 P a48f1da8eee34813ba07200ee0415c41 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:30:22.990175 29151 raft_consensus.cc:482] T 00000000000000000000000000000000 P a48f1da8eee34813ba07200ee0415c41 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:30:22.990227 29151 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a48f1da8eee34813ba07200ee0415c41 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:22.990772 29151 raft_consensus.cc:504] T 00000000000000000000000000000000 P a48f1da8eee34813ba07200ee0415c41 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48f1da8eee34813ba07200ee0415c41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37379 } }
I20260618 23:30:22.990899 29151 leader_election.cc:302] T 00000000000000000000000000000000 P a48f1da8eee34813ba07200ee0415c41 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a48f1da8eee34813ba07200ee0415c41; no voters:
I20260618 23:30:22.991087 29151 leader_election.cc:288] T 00000000000000000000000000000000 P a48f1da8eee34813ba07200ee0415c41 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:30:22.991204 29154 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a48f1da8eee34813ba07200ee0415c41 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:30:22.991346 29151 sys_catalog.cc:564] T 00000000000000000000000000000000 P a48f1da8eee34813ba07200ee0415c41 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:30:22.991400 29154 raft_consensus.cc:686] T 00000000000000000000000000000000 P a48f1da8eee34813ba07200ee0415c41 [term 1 LEADER]: Becoming Leader. State: Replica: a48f1da8eee34813ba07200ee0415c41, State: Running, Role: LEADER
I20260618 23:30:22.991524 29154 consensus_queue.cc:237] T 00000000000000000000000000000000 P a48f1da8eee34813ba07200ee0415c41 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48f1da8eee34813ba07200ee0415c41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37379 } }
I20260618 23:30:22.991544 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 29083
I20260618 23:30:22.991633 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:30:22.991801 8350 external_mini_cluster.cc:1428] Setting key d70dc6c2556d8c45a22c83a3814b5543
I20260618 23:30:22.992092 29156 sys_catalog.cc:455] T 00000000000000000000000000000000 P a48f1da8eee34813ba07200ee0415c41 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a48f1da8eee34813ba07200ee0415c41. Latest consensus state: current_term: 1 leader_uuid: "a48f1da8eee34813ba07200ee0415c41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48f1da8eee34813ba07200ee0415c41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37379 } } }
I20260618 23:30:22.992179 29156 sys_catalog.cc:458] T 00000000000000000000000000000000 P a48f1da8eee34813ba07200ee0415c41 [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:22.992202 29155 sys_catalog.cc:455] T 00000000000000000000000000000000 P a48f1da8eee34813ba07200ee0415c41 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a48f1da8eee34813ba07200ee0415c41" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a48f1da8eee34813ba07200ee0415c41" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37379 } } }
I20260618 23:30:22.992252 29155 sys_catalog.cc:458] T 00000000000000000000000000000000 P a48f1da8eee34813ba07200ee0415c41 [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:22.993052 29164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:30:22.993722 29164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:30:22.995265 29164 catalog_manager.cc:1347] Generated new cluster ID: 2e26c6b2f7304de69c4c5cf1ad78e4aa
I20260618 23:30:22.995312 29164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 18 23:30:22 dist-test-slave-47qt krb5kdc[29070](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260618 23:30:23.004240 29166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:22.992487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:58621 (local address 127.0.0.1:37379)
0618 23:30:22.992656 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:22.992666 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:22.992700 (+ 34us) server_negotiation.cc:407] Connection header received
0618 23:30:22.992857 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:22.992864 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:22.992942 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:22.993923 (+ 981us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:22.994011 (+ 88us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:22.995181 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:22.996308 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:22.996909 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:22.997004 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:22.999853 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:22.999857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:22.999858 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:22.999871 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:23.002064 (+ 2193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:23.002514 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:23.002517 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:23.002518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:23.002648 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:23.002886 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:23.002887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:23.002887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:23.003223 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:23.003406 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:23.003535 (+ 129us) server_negotiation.cc:299] Negotiation successful
0618 23:30:23.003638 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":74,"threads_started":1}
I20260618 23:30:23.005702 8350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:30:23.015578 29166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:23.007154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:40097 (local address 127.0.0.1:37379)
0618 23:30:23.007199 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:23.007200 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:23.007282 (+ 82us) server_negotiation.cc:407] Connection header received
0618 23:30:23.007331 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:23.007332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:23.007343 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:23.007392 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:23.008014 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:23.008746 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:23.009907 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:23.010223 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:23.010282 (+ 59us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:23.011752 (+ 1470us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:23.011754 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:23.011754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:23.011761 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:23.013679 (+ 1918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:23.014376 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:23.014377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:23.014378 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:23.014457 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:23.014905 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:23.014907 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:23.014908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:23.015061 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:23.015194 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:23.015376 (+ 182us) server_negotiation.cc:299] Negotiation successful
0618 23:30:23.015455 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260618 23:30:23.030261 29164 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:30:23.030803 29164 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:30:23.038937 29164 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a48f1da8eee34813ba07200ee0415c41: Generated new TSK 0
I20260618 23:30:23.039300 29164 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:30:23.045258 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 29083
W20260618 23:30:23.049631 29068 connection.cc:535] client connection to 127.0.0.1:37379 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37379): Connection reset by peer (error 104)
2026-06-18T23:30:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1394 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:30:23 dist-test-slave-47qt krb5kdc[29182](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:30:23 dist-test-slave-47qt krb5kdc[29182](info): set up 2 sockets
Jun 18 23:30:23 dist-test-slave-47qt krb5kdc[29182](info): commencing operation
krb5kdc: starting...
W20260618 23:30:24.109573 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:30:24 dist-test-slave-47qt krb5kdc[29182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:30:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:30:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 23:30:24.240324 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42951
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:30:24.325264 29195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:24.325542 29195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:24.325587 29195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:24.326812 29195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:30:24.326889 29195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:24.326915 29195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:24.326943 29195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:30:24.326982 29195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:30:24.328514 29195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42951
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:24.328819 29195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:24.329077 29195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:24.335559 29201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:24.335546 29200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:24.335570 29195 server_base.cc:1034] running on GCE node
W20260618 23:30:24.335548 29203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:24.336109 29195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:24.336433 29195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:24.337628 29195 hybrid_clock.cc:648] HybridClock initialized: now 1781825424337611 us; error 46 us; skew 500 ppm
Jun 18 23:30:24 dist-test-slave-47qt krb5kdc[29182](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825424, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:24.340345 29195 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 23:30:24.340850 29195 webserver.cc:460] Webserver started at http://127.0.0.1:34525/ using document root <none> and password file <none>
I20260618 23:30:24.341089 29195 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:24.341179 29195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:24.341343 29195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:24.342424 29195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "420620268d784ade8ab8a6b507f4a2c1"
format_stamp: "Formatted at 2026-06-18 23:30:24 on dist-test-slave-47qt"
server_key: "c62c22c7b71d23b2d5fa60b07b4ddca8"
server_key_iv: "99d94406e04bfc4e3e014619ee45061b"
server_key_version: "encryptionkey@0"
I20260618 23:30:24.342797 29195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "420620268d784ade8ab8a6b507f4a2c1"
format_stamp: "Formatted at 2026-06-18 23:30:24 on dist-test-slave-47qt"
server_key: "c62c22c7b71d23b2d5fa60b07b4ddca8"
server_key_iv: "99d94406e04bfc4e3e014619ee45061b"
server_key_version: "encryptionkey@0"
I20260618 23:30:24.344391 29195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:30:24.345470 29210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:24.345804 29195 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:30:24.346000 29195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "420620268d784ade8ab8a6b507f4a2c1"
format_stamp: "Formatted at 2026-06-18 23:30:24 on dist-test-slave-47qt"
server_key: "c62c22c7b71d23b2d5fa60b07b4ddca8"
server_key_iv: "99d94406e04bfc4e3e014619ee45061b"
server_key_version: "encryptionkey@0"
I20260618 23:30:24.346117 29195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:24.365885 29195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:24.367993 29195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:24.368170 29195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:24.372671 29195 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42951
I20260618 23:30:24.372707 29262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42951 every 8 connection(s)
I20260618 23:30:24.373080 29195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:30:24.373987 29263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:24.375558 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 29195
I20260618 23:30:24.375715 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:30:24.375941 8350 external_mini_cluster.cc:1428] Setting key ec0608ed9d370998ffd04a9a5167f682
I20260618 23:30:24.376740 29263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 420620268d784ade8ab8a6b507f4a2c1: Bootstrap starting.
I20260618 23:30:24.377439 29263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 420620268d784ade8ab8a6b507f4a2c1: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:24.377794 29263 log.cc:824] T 00000000000000000000000000000000 P 420620268d784ade8ab8a6b507f4a2c1: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:24.378587 29263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 420620268d784ade8ab8a6b507f4a2c1: No bootstrap required, opened a new log
I20260618 23:30:24.381239 29263 raft_consensus.cc:348] T 00000000000000000000000000000000 P 420620268d784ade8ab8a6b507f4a2c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "420620268d784ade8ab8a6b507f4a2c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42951 } }
I20260618 23:30:24.381450 29263 raft_consensus.cc:374] T 00000000000000000000000000000000 P 420620268d784ade8ab8a6b507f4a2c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:24.381510 29263 raft_consensus.cc:729] T 00000000000000000000000000000000 P 420620268d784ade8ab8a6b507f4a2c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 420620268d784ade8ab8a6b507f4a2c1, State: Initialized, Role: FOLLOWER
I20260618 23:30:24.381675 29263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 420620268d784ade8ab8a6b507f4a2c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "420620268d784ade8ab8a6b507f4a2c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42951 } }
I20260618 23:30:24.381752 29263 raft_consensus.cc:388] T 00000000000000000000000000000000 P 420620268d784ade8ab8a6b507f4a2c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:30:24.381775 29263 raft_consensus.cc:482] T 00000000000000000000000000000000 P 420620268d784ade8ab8a6b507f4a2c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:30:24.381846 29263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 420620268d784ade8ab8a6b507f4a2c1 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:24.382320 29266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:24.376596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:33165 (local address 127.0.0.1:42951)
0618 23:30:24.376786 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:24.376820 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:24.376853 (+ 33us) server_negotiation.cc:407] Connection header received
0618 23:30:24.377354 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:24.377360 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:24.377409 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:24.377567 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:24.377834 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:24.378942 (+ 1108us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:24.380148 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:24.380538 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:24.380617 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:24.381535 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:24.381539 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:24.381540 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:24.381781 (+ 241us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:24.381821 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:24.381843 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:24.381925 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:24.382092 (+ 167us) server_negotiation.cc:299] Negotiation successful
0618 23:30:24.382122 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":86,"threads_started":1}
I20260618 23:30:24.382561 29263 raft_consensus.cc:504] T 00000000000000000000000000000000 P 420620268d784ade8ab8a6b507f4a2c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "420620268d784ade8ab8a6b507f4a2c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42951 } }
I20260618 23:30:24.382673 29263 leader_election.cc:302] T 00000000000000000000000000000000 P 420620268d784ade8ab8a6b507f4a2c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 420620268d784ade8ab8a6b507f4a2c1; no voters:
I20260618 23:30:24.382834 29263 leader_election.cc:288] T 00000000000000000000000000000000 P 420620268d784ade8ab8a6b507f4a2c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:30:24.382853 29268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 420620268d784ade8ab8a6b507f4a2c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:30:24.383072 29268 raft_consensus.cc:686] T 00000000000000000000000000000000 P 420620268d784ade8ab8a6b507f4a2c1 [term 1 LEADER]: Becoming Leader. State: Replica: 420620268d784ade8ab8a6b507f4a2c1, State: Running, Role: LEADER
I20260618 23:30:24.383219 8350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:30:24.383198 29263 sys_catalog.cc:564] T 00000000000000000000000000000000 P 420620268d784ade8ab8a6b507f4a2c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:30:24.383198 29268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 420620268d784ade8ab8a6b507f4a2c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "420620268d784ade8ab8a6b507f4a2c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42951 } }
I20260618 23:30:24.384426 29270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 420620268d784ade8ab8a6b507f4a2c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 420620268d784ade8ab8a6b507f4a2c1. Latest consensus state: current_term: 1 leader_uuid: "420620268d784ade8ab8a6b507f4a2c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "420620268d784ade8ab8a6b507f4a2c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42951 } } }
I20260618 23:30:24.384517 29270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 420620268d784ade8ab8a6b507f4a2c1 [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:24.384689 29269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 420620268d784ade8ab8a6b507f4a2c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "420620268d784ade8ab8a6b507f4a2c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "420620268d784ade8ab8a6b507f4a2c1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42951 } } }
I20260618 23:30:24.384979 29269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 420620268d784ade8ab8a6b507f4a2c1 [sys.catalog]: This master's current role is: LEADER
W20260618 23:30:24.385154 29288 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 420620268d784ade8ab8a6b507f4a2c1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 23:30:24.385203 29288 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:30:24.385267 29287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:30:24.385479 29287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:30:24.386876 29287 catalog_manager.cc:1347] Generated new cluster ID: 5083319351c7402884b8d1d56ab8a8c6
I20260618 23:30:24.386922 29287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:30:24.388947 29266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:24.385043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:57881 (local address 127.0.0.1:42951)
0618 23:30:24.385076 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:24.385078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:24.385367 (+ 289us) server_negotiation.cc:407] Connection header received
0618 23:30:24.385457 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:24.385459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:24.385470 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:24.385546 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:24.385695 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:24.386517 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:24.387613 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:24.387919 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:24.387994 (+ 75us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:24.388700 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:24.388701 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:24.388701 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:24.388705 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:24.388718 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:24.388722 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:24.388769 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:24.388869 (+ 100us) server_negotiation.cc:299] Negotiation successful
0618 23:30:24.388879 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260618 23:30:24.396751 29287 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:30:24.397405 29287 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:30:24.401022 29287 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 420620268d784ade8ab8a6b507f4a2c1: Generated new TSK 0
I20260618 23:30:24.401243 29287 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:30:24.404778 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 29195
W20260618 23:30:24.409093 29180 connection.cc:535] client connection to 127.0.0.1:42951 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42951): Connection reset by peer (error 104)
2026-06-18T23:30:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1359 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:30:24 dist-test-slave-47qt krb5kdc[29294](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:30:24 dist-test-slave-47qt krb5kdc[29294](info): set up 2 sockets
Jun 18 23:30:24 dist-test-slave-47qt krb5kdc[29294](info): commencing operation
krb5kdc: starting...
W20260618 23:30:25.453636 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:30:25 dist-test-slave-47qt krb5kdc[29294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825425, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:30:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:30:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260618 23:30:25.580994 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38539
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:30:25.675168 29307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:25.675387 29307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:25.675411 29307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:25.676646 29307 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:30:25.676693 29307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:25.676708 29307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:25.676721 29307 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:30:25.676733 29307 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:30:25.678195 29307 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38539
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825053542692-8350-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:25.678390 29307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:25.678563 29307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:25.683445 29312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:25.683454 29313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:25.683490 29315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:25.683621 29307 server_base.cc:1034] running on GCE node
I20260618 23:30:25.683938 29307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:25.684242 29307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:25.685406 29307 hybrid_clock.cc:648] HybridClock initialized: now 1781825425685387 us; error 29 us; skew 500 ppm
Jun 18 23:30:25 dist-test-slave-47qt krb5kdc[29294](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825425, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:25.687768 29307 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260618 23:30:25.688179 29307 webserver.cc:460] Webserver started at http://127.0.0.1:44585/ using document root <none> and password file <none>
I20260618 23:30:25.688418 29307 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:25.688485 29307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:25.688637 29307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:25.689729 29307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "813fb201da09418b898ed1733bba2c3f"
format_stamp: "Formatted at 2026-06-18 23:30:25 on dist-test-slave-47qt"
server_key: "48cb5b2e114a87598f26a129c3c2738b"
server_key_iv: "24297b7f6cccfd96d8ba355dfd2e440a"
server_key_version: "encryptionkey@0"
I20260618 23:30:25.690078 29307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "813fb201da09418b898ed1733bba2c3f"
format_stamp: "Formatted at 2026-06-18 23:30:25 on dist-test-slave-47qt"
server_key: "48cb5b2e114a87598f26a129c3c2738b"
server_key_iv: "24297b7f6cccfd96d8ba355dfd2e440a"
server_key_version: "encryptionkey@0"
I20260618 23:30:25.691689 29307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:30:25.692587 29322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:25.692773 29307 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260618 23:30:25.692868 29307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "813fb201da09418b898ed1733bba2c3f"
format_stamp: "Formatted at 2026-06-18 23:30:25 on dist-test-slave-47qt"
server_key: "48cb5b2e114a87598f26a129c3c2738b"
server_key_iv: "24297b7f6cccfd96d8ba355dfd2e440a"
server_key_version: "encryptionkey@0"
I20260618 23:30:25.692973 29307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:25.696120 29307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:25.698002 29307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:25.698159 29307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:25.702589 29307 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38539
I20260618 23:30:25.702778 29374 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38539 every 8 connection(s)
I20260618 23:30:25.702999 29307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:30:25.704042 29375 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:25.706218 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 29307
I20260618 23:30:25.706339 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:30:25.706534 8350 external_mini_cluster.cc:1428] Setting key 62e171043b60ad73a50c8b03e9e859a1
I20260618 23:30:25.706799 29375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 813fb201da09418b898ed1733bba2c3f: Bootstrap starting.
I20260618 23:30:25.707578 29375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 813fb201da09418b898ed1733bba2c3f: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:25.707871 29375 log.cc:824] T 00000000000000000000000000000000 P 813fb201da09418b898ed1733bba2c3f: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:25.708668 29375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 813fb201da09418b898ed1733bba2c3f: No bootstrap required, opened a new log
I20260618 23:30:25.709798 29378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:25.707132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:58101 (local address 127.0.0.1:38539)
0618 23:30:25.707290 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:25.707298 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:25.707331 (+ 33us) server_negotiation.cc:407] Connection header received
0618 23:30:25.707799 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:25.707806 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:25.707830 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:25.707972 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:25.708598 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:25.708602 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:25.708602 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:25.708606 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:25.708648 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:25.708661 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:25.708758 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:25.709562 (+ 804us) server_negotiation.cc:299] Negotiation successful
0618 23:30:25.709606 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":80,"threads_started":1}
I20260618 23:30:25.710919 29375 raft_consensus.cc:348] T 00000000000000000000000000000000 P 813fb201da09418b898ed1733bba2c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "813fb201da09418b898ed1733bba2c3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38539 } }
I20260618 23:30:25.711078 29375 raft_consensus.cc:374] T 00000000000000000000000000000000 P 813fb201da09418b898ed1733bba2c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:25.711118 8350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:30:25.711140 29375 raft_consensus.cc:729] T 00000000000000000000000000000000 P 813fb201da09418b898ed1733bba2c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 813fb201da09418b898ed1733bba2c3f, State: Initialized, Role: FOLLOWER
I20260618 23:30:25.711252 29375 consensus_queue.cc:260] T 00000000000000000000000000000000 P 813fb201da09418b898ed1733bba2c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "813fb201da09418b898ed1733bba2c3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38539 } }
I20260618 23:30:25.711331 29375 raft_consensus.cc:388] T 00000000000000000000000000000000 P 813fb201da09418b898ed1733bba2c3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:30:25.711357 29375 raft_consensus.cc:482] T 00000000000000000000000000000000 P 813fb201da09418b898ed1733bba2c3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:30:25.711392 29375 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 813fb201da09418b898ed1733bba2c3f [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:25.712112 29375 raft_consensus.cc:504] T 00000000000000000000000000000000 P 813fb201da09418b898ed1733bba2c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "813fb201da09418b898ed1733bba2c3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38539 } }
I20260618 23:30:25.712232 29375 leader_election.cc:302] T 00000000000000000000000000000000 P 813fb201da09418b898ed1733bba2c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 813fb201da09418b898ed1733bba2c3f; no voters:
I20260618 23:30:25.712448 29375 leader_election.cc:288] T 00000000000000000000000000000000 P 813fb201da09418b898ed1733bba2c3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:30:25.712467 29384 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 813fb201da09418b898ed1733bba2c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:30:25.712625 29384 raft_consensus.cc:686] T 00000000000000000000000000000000 P 813fb201da09418b898ed1733bba2c3f [term 1 LEADER]: Becoming Leader. State: Replica: 813fb201da09418b898ed1733bba2c3f, State: Running, Role: LEADER
I20260618 23:30:25.712703 29375 sys_catalog.cc:564] T 00000000000000000000000000000000 P 813fb201da09418b898ed1733bba2c3f [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:30:25.712711 29384 consensus_queue.cc:237] T 00000000000000000000000000000000 P 813fb201da09418b898ed1733bba2c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "813fb201da09418b898ed1733bba2c3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38539 } }
I20260618 23:30:25.713073 29384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 813fb201da09418b898ed1733bba2c3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 813fb201da09418b898ed1733bba2c3f. Latest consensus state: current_term: 1 leader_uuid: "813fb201da09418b898ed1733bba2c3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "813fb201da09418b898ed1733bba2c3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38539 } } }
I20260618 23:30:25.713229 29384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 813fb201da09418b898ed1733bba2c3f [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:25.713492 29385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 813fb201da09418b898ed1733bba2c3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "813fb201da09418b898ed1733bba2c3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "813fb201da09418b898ed1733bba2c3f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38539 } } }
I20260618 23:30:25.713563 29385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 813fb201da09418b898ed1733bba2c3f [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:25.714078 29393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:30:25.714289 29378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:25.712806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.63:41941 (local address 127.0.0.1:38539)
0618 23:30:25.712844 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:25.712846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:25.713027 (+ 181us) server_negotiation.cc:407] Connection header received
0618 23:30:25.713092 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:25.713093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:25.713099 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:25.713225 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:25.713890 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:25.713891 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:25.713891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:25.713895 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:25.713909 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:25.713913 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:25.713976 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:25.714218 (+ 242us) server_negotiation.cc:299] Negotiation successful
0618 23:30:25.714232 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260618 23:30:25.714955 29393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:30:25.716593 29393 catalog_manager.cc:1347] Generated new cluster ID: acdd09d3a7ae405b919644cf972b993d
I20260618 23:30:25.716650 29393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:30:25.733457 29393 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:30:25.733649 29393 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:30:25.743526 29393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 813fb201da09418b898ed1733bba2c3f: Generated new TSK 0
I20260618 23:30:25.743746 29393 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:30:25.753861 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 29307
2026-06-18T23:30:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:30:25 dist-test-slave-47qt krb5kdc[29406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:30:25 dist-test-slave-47qt krb5kdc[29406](info): set up 2 sockets
Jun 18 23:30:25 dist-test-slave-47qt krb5kdc[29406](info): commencing operation
krb5kdc: starting...
W20260618 23:30:26.797209 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:30:26 dist-test-slave-47qt krb5kdc[29406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:30:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:30:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.39.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
I20260618 23:30:26.918069 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:33367
--webserver_interface=127.8.39.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:44039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:33367
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:30:26.997678 29419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:26.997884 29419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:26.997907 29419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:26.999069 29419 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:30:26.999112 29419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:26.999126 29419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:26.999140 29419 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:30:26.999197 29419 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:30:27.000625 29419 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:44039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:33367
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:33367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:27.000880 29419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:27.001104 29419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:27.006049 29425 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:27.006111 29427 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:27.006052 29424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:27.006372 29419 server_base.cc:1034] running on GCE node
I20260618 23:30:27.006620 29419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:27.006893 29419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:27.008057 29419 hybrid_clock.cc:648] HybridClock initialized: now 1781825427008040 us; error 28 us; skew 500 ppm
Jun 18 23:30:27 dist-test-slave-47qt krb5kdc[29406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825427, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:27.017154 29419 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:30:27.017614 29419 webserver.cc:460] Webserver started at http://127.8.39.190:44485/ using document root <none> and password file <none>
I20260618 23:30:27.017848 29419 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:27.017916 29419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:27.018076 29419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:27.019184 29419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "cb6cbc3ea7b34347b9497dbcbca78549"
format_stamp: "Formatted at 2026-06-18 23:30:27 on dist-test-slave-47qt"
server_key: "a403feaafac8a4c3168c3222cb5d9142"
server_key_iv: "8d9b0c540de206c09b84aa8957430e5a"
server_key_version: "encryptionkey@0"
I20260618 23:30:27.019563 29419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "cb6cbc3ea7b34347b9497dbcbca78549"
format_stamp: "Formatted at 2026-06-18 23:30:27 on dist-test-slave-47qt"
server_key: "a403feaafac8a4c3168c3222cb5d9142"
server_key_iv: "8d9b0c540de206c09b84aa8957430e5a"
server_key_version: "encryptionkey@0"
I20260618 23:30:27.020918 29419 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:30:27.021777 29434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:27.021979 29419 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:30:27.022081 29419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "cb6cbc3ea7b34347b9497dbcbca78549"
format_stamp: "Formatted at 2026-06-18 23:30:27 on dist-test-slave-47qt"
server_key: "a403feaafac8a4c3168c3222cb5d9142"
server_key_iv: "8d9b0c540de206c09b84aa8957430e5a"
server_key_version: "encryptionkey@0"
I20260618 23:30:27.022130 29419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:27.040665 29419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:27.043006 29419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:27.043155 29419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:27.048045 29419 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:33367
I20260618 23:30:27.048146 29486 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:33367 every 8 connection(s)
I20260618 23:30:27.048491 29419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:30:27.049451 29487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:27.051689 29487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb6cbc3ea7b34347b9497dbcbca78549: Bootstrap starting.
I20260618 23:30:27.052381 29487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cb6cbc3ea7b34347b9497dbcbca78549: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:27.052698 29487 log.cc:824] T 00000000000000000000000000000000 P cb6cbc3ea7b34347b9497dbcbca78549: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:27.052848 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 29419
I20260618 23:30:27.052973 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:30:27.053205 8350 external_mini_cluster.cc:1428] Setting key 8e29d480d0e28ee93ca61808e177bb68
I20260618 23:30:27.053496 29487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cb6cbc3ea7b34347b9497dbcbca78549: No bootstrap required, opened a new log
I20260618 23:30:27.055527 29487 raft_consensus.cc:348] T 00000000000000000000000000000000 P cb6cbc3ea7b34347b9497dbcbca78549 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6cbc3ea7b34347b9497dbcbca78549" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33367 } }
I20260618 23:30:27.055735 29487 raft_consensus.cc:374] T 00000000000000000000000000000000 P cb6cbc3ea7b34347b9497dbcbca78549 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:27.055779 29487 raft_consensus.cc:729] T 00000000000000000000000000000000 P cb6cbc3ea7b34347b9497dbcbca78549 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cb6cbc3ea7b34347b9497dbcbca78549, State: Initialized, Role: FOLLOWER
I20260618 23:30:27.055933 29487 consensus_queue.cc:260] T 00000000000000000000000000000000 P cb6cbc3ea7b34347b9497dbcbca78549 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6cbc3ea7b34347b9497dbcbca78549" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33367 } }
I20260618 23:30:27.056033 29487 raft_consensus.cc:388] T 00000000000000000000000000000000 P cb6cbc3ea7b34347b9497dbcbca78549 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:30:27.056073 29487 raft_consensus.cc:482] T 00000000000000000000000000000000 P cb6cbc3ea7b34347b9497dbcbca78549 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:30:27.056141 29487 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cb6cbc3ea7b34347b9497dbcbca78549 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:27.057320 29487 raft_consensus.cc:504] T 00000000000000000000000000000000 P cb6cbc3ea7b34347b9497dbcbca78549 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6cbc3ea7b34347b9497dbcbca78549" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33367 } }
I20260618 23:30:27.057447 29487 leader_election.cc:302] T 00000000000000000000000000000000 P cb6cbc3ea7b34347b9497dbcbca78549 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cb6cbc3ea7b34347b9497dbcbca78549; no voters:
I20260618 23:30:27.057636 29487 leader_election.cc:288] T 00000000000000000000000000000000 P cb6cbc3ea7b34347b9497dbcbca78549 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:30:27.057662 29492 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cb6cbc3ea7b34347b9497dbcbca78549 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:30:27.057780 29492 raft_consensus.cc:686] T 00000000000000000000000000000000 P cb6cbc3ea7b34347b9497dbcbca78549 [term 1 LEADER]: Becoming Leader. State: Replica: cb6cbc3ea7b34347b9497dbcbca78549, State: Running, Role: LEADER
I20260618 23:30:27.057920 29487 sys_catalog.cc:564] T 00000000000000000000000000000000 P cb6cbc3ea7b34347b9497dbcbca78549 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:30:27.057925 29492 consensus_queue.cc:237] T 00000000000000000000000000000000 P cb6cbc3ea7b34347b9497dbcbca78549 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6cbc3ea7b34347b9497dbcbca78549" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33367 } }
I20260618 23:30:27.058349 29493 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb6cbc3ea7b34347b9497dbcbca78549 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cb6cbc3ea7b34347b9497dbcbca78549" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6cbc3ea7b34347b9497dbcbca78549" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33367 } } }
I20260618 23:30:27.058568 29493 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb6cbc3ea7b34347b9497dbcbca78549 [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:27.058494 29494 sys_catalog.cc:455] T 00000000000000000000000000000000 P cb6cbc3ea7b34347b9497dbcbca78549 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cb6cbc3ea7b34347b9497dbcbca78549. Latest consensus state: current_term: 1 leader_uuid: "cb6cbc3ea7b34347b9497dbcbca78549" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cb6cbc3ea7b34347b9497dbcbca78549" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33367 } } }
I20260618 23:30:27.058807 29494 sys_catalog.cc:458] T 00000000000000000000000000000000 P cb6cbc3ea7b34347b9497dbcbca78549 [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:27.059150 29499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:30:27.059608 29499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:30:27.061435 29499 catalog_manager.cc:1347] Generated new cluster ID: b475167e232f40049b81b0fbcdb42c93
I20260618 23:30:27.061493 29499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 18 23:30:27 dist-test-slave-47qt krb5kdc[29406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825426, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:30:27.067569 29491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:27.053699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55825 (local address 127.8.39.190:33367)
0618 23:30:27.053870 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:27.053885 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:27.053913 (+ 28us) server_negotiation.cc:407] Connection header received
0618 23:30:27.055508 (+ 1595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:27.055515 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:27.055570 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:27.055700 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:27.056646 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:27.058066 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:27.059174 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:27.059770 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:27.060247 (+ 477us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:27.063320 (+ 3073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:27.063324 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:27.063325 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:27.063336 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:27.065476 (+ 2140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:27.065931 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:27.065934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:27.065934 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:27.066049 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:27.066290 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:27.066291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:27.066291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:27.066566 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:27.067215 (+ 649us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:27.067288 (+ 73us) server_negotiation.cc:299] Negotiation successful
0618 23:30:27.067381 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":97,"threads_started":1}
I20260618 23:30:27.068593 8350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:30:27.071363 29499 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:30:27.072288 29499 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:30:27.076609 29491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:27.070010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42105 (local address 127.8.39.190:33367)
0618 23:30:27.070042 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:27.070043 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:27.070139 (+ 96us) server_negotiation.cc:407] Connection header received
0618 23:30:27.070223 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:27.070224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:27.070231 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:27.070256 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:27.070935 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:27.071620 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:27.072366 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:27.072608 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:27.072706 (+ 98us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:27.073856 (+ 1150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:27.073857 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:27.073858 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:27.073864 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:27.075432 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:27.075810 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:27.075811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:27.075811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:27.075870 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:27.076093 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:27.076094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:27.076094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:27.076202 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:27.076313 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:27.076440 (+ 127us) server_negotiation.cc:299] Negotiation successful
0618 23:30:27.076503 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260618 23:30:27.086158 29499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cb6cbc3ea7b34347b9497dbcbca78549: Generated new TSK 0
I20260618 23:30:27.086431 29499 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:30:27.095845 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 29419
W20260618 23:30:27.100159 29404 connection.cc:535] client connection to 127.8.39.190:33367 recv error: Network error: failed to read from TLS socket (remote: 127.8.39.190:33367): Connection reset by peer (error 104)
2026-06-18T23:30:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1341 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:30:27 dist-test-slave-47qt krb5kdc[29518](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:30:27 dist-test-slave-47qt krb5kdc[29518](info): set up 2 sockets
Jun 18 23:30:27 dist-test-slave-47qt krb5kdc[29518](info): commencing operation
krb5kdc: starting...
W20260618 23:30:28.145301 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:30:28 dist-test-slave-47qt krb5kdc[29518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825428, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:30:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:30:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.39.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
I20260618 23:30:28.269665 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:42975
--webserver_interface=127.8.39.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:43455
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:42975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:30:28.351501 29531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:28.351780 29531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:28.351819 29531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:28.352952 29531 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:30:28.353065 29531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:28.353096 29531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:28.353168 29531 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:30:28.353200 29531 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:30:28.354668 29531 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:43455
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:42975
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:42975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29531
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:28.354947 29531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:28.355185 29531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:28.360284 29537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:28.360284 29539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:28.360287 29536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:28.360520 29531 server_base.cc:1034] running on GCE node
I20260618 23:30:28.360870 29531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:28.361400 29531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:28.362612 29531 hybrid_clock.cc:648] HybridClock initialized: now 1781825428362586 us; error 47 us; skew 500 ppm
Jun 18 23:30:28 dist-test-slave-47qt krb5kdc[29518](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825428, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:28.365430 29531 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:30:28.365917 29531 webserver.cc:460] Webserver started at http://127.8.39.190:34421/ using document root <none> and password file <none>
I20260618 23:30:28.366154 29531 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:28.366199 29531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:28.366343 29531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:28.367362 29531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "75a33fcd14da4a8890543fd3a5ef187f"
format_stamp: "Formatted at 2026-06-18 23:30:28 on dist-test-slave-47qt"
server_key: "ea03218fd7c0887996963b1db7e788dd"
server_key_iv: "add4c278758358c4890d2876d0c3393b"
server_key_version: "encryptionkey@0"
I20260618 23:30:28.367712 29531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "75a33fcd14da4a8890543fd3a5ef187f"
format_stamp: "Formatted at 2026-06-18 23:30:28 on dist-test-slave-47qt"
server_key: "ea03218fd7c0887996963b1db7e788dd"
server_key_iv: "add4c278758358c4890d2876d0c3393b"
server_key_version: "encryptionkey@0"
I20260618 23:30:28.368996 29531 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:30:28.369879 29546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:28.370163 29531 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:30:28.370258 29531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "75a33fcd14da4a8890543fd3a5ef187f"
format_stamp: "Formatted at 2026-06-18 23:30:28 on dist-test-slave-47qt"
server_key: "ea03218fd7c0887996963b1db7e788dd"
server_key_iv: "add4c278758358c4890d2876d0c3393b"
server_key_version: "encryptionkey@0"
I20260618 23:30:28.370352 29531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:28.387188 29531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:28.389683 29531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:28.389801 29531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:28.394014 29531 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:42975
I20260618 23:30:28.394181 29598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:42975 every 8 connection(s)
I20260618 23:30:28.394443 29531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:30:28.395453 29599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:28.397799 29599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75a33fcd14da4a8890543fd3a5ef187f: Bootstrap starting.
I20260618 23:30:28.398368 29599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75a33fcd14da4a8890543fd3a5ef187f: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:28.398656 29599 log.cc:824] T 00000000000000000000000000000000 P 75a33fcd14da4a8890543fd3a5ef187f: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:28.399426 29599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75a33fcd14da4a8890543fd3a5ef187f: No bootstrap required, opened a new log
I20260618 23:30:28.401367 29599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 75a33fcd14da4a8890543fd3a5ef187f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a33fcd14da4a8890543fd3a5ef187f" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 42975 } }
I20260618 23:30:28.401535 29599 raft_consensus.cc:374] T 00000000000000000000000000000000 P 75a33fcd14da4a8890543fd3a5ef187f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:28.401578 29599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 75a33fcd14da4a8890543fd3a5ef187f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75a33fcd14da4a8890543fd3a5ef187f, State: Initialized, Role: FOLLOWER
I20260618 23:30:28.401757 29599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75a33fcd14da4a8890543fd3a5ef187f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a33fcd14da4a8890543fd3a5ef187f" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 42975 } }
I20260618 23:30:28.401837 29599 raft_consensus.cc:388] T 00000000000000000000000000000000 P 75a33fcd14da4a8890543fd3a5ef187f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:30:28.401877 29599 raft_consensus.cc:482] T 00000000000000000000000000000000 P 75a33fcd14da4a8890543fd3a5ef187f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:30:28.401969 29599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 75a33fcd14da4a8890543fd3a5ef187f [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:28.402823 29599 raft_consensus.cc:504] T 00000000000000000000000000000000 P 75a33fcd14da4a8890543fd3a5ef187f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a33fcd14da4a8890543fd3a5ef187f" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 42975 } }
I20260618 23:30:28.402973 29599 leader_election.cc:302] T 00000000000000000000000000000000 P 75a33fcd14da4a8890543fd3a5ef187f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75a33fcd14da4a8890543fd3a5ef187f; no voters:
I20260618 23:30:28.403163 29599 leader_election.cc:288] T 00000000000000000000000000000000 P 75a33fcd14da4a8890543fd3a5ef187f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:30:28.403199 29602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 75a33fcd14da4a8890543fd3a5ef187f [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:30:28.403347 29602 raft_consensus.cc:686] T 00000000000000000000000000000000 P 75a33fcd14da4a8890543fd3a5ef187f [term 1 LEADER]: Becoming Leader. State: Replica: 75a33fcd14da4a8890543fd3a5ef187f, State: Running, Role: LEADER
I20260618 23:30:28.403420 29599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 75a33fcd14da4a8890543fd3a5ef187f [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:30:28.403492 29602 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75a33fcd14da4a8890543fd3a5ef187f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a33fcd14da4a8890543fd3a5ef187f" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 42975 } }
I20260618 23:30:28.403887 29604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75a33fcd14da4a8890543fd3a5ef187f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75a33fcd14da4a8890543fd3a5ef187f. Latest consensus state: current_term: 1 leader_uuid: "75a33fcd14da4a8890543fd3a5ef187f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a33fcd14da4a8890543fd3a5ef187f" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 42975 } } }
I20260618 23:30:28.403982 29604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75a33fcd14da4a8890543fd3a5ef187f [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:28.403874 29603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75a33fcd14da4a8890543fd3a5ef187f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75a33fcd14da4a8890543fd3a5ef187f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75a33fcd14da4a8890543fd3a5ef187f" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 42975 } } }
I20260618 23:30:28.404345 29603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75a33fcd14da4a8890543fd3a5ef187f [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:28.404489 29610 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:30:28.404721 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 29531
I20260618 23:30:28.404836 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:30:28.405022 29610 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:30:28.405401 8350 external_mini_cluster.cc:1428] Setting key c0290ba5fdeaa253bcbc11379dcda2f7
I20260618 23:30:28.407905 29610 catalog_manager.cc:1347] Generated new cluster ID: 4f82939b19db471eb0b210876c8edb80
I20260618 23:30:28.407963 29610 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:30:28.412953 29615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:28.405920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57101 (local address 127.8.39.190:42975)
0618 23:30:28.406188 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:28.406196 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:28.406219 (+ 23us) server_negotiation.cc:407] Connection header received
0618 23:30:28.406306 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:28.406311 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:28.406353 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:28.406489 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:28.406685 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:28.409451 (+ 2766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:28.410653 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:28.411273 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:28.411381 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:28.412215 (+ 834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:28.412220 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:28.412220 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:28.412427 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:28.412478 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:28.412493 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:28.412581 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:28.412719 (+ 138us) server_negotiation.cc:299] Negotiation successful
0618 23:30:28.412746 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":219,"thread_start_us":65,"threads_started":1}
I20260618 23:30:28.414007 8350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:30:28.414618 29610 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:30:28.415333 29610 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:30:28.419294 29615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:28.415563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44199 (local address 127.8.39.190:42975)
0618 23:30:28.415594 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:28.415595 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:28.415734 (+ 139us) server_negotiation.cc:407] Connection header received
0618 23:30:28.415777 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:28.415778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:28.415787 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:28.415846 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:28.416090 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:28.416833 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:28.417946 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:28.418254 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:28.418314 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:28.419034 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:28.419035 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:28.419036 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:28.419040 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:28.419055 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:28.419059 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:28.419113 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:28.419208 (+ 95us) server_negotiation.cc:299] Negotiation successful
0618 23:30:28.419220 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260618 23:30:28.420533 29610 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 75a33fcd14da4a8890543fd3a5ef187f: Generated new TSK 0
I20260618 23:30:28.420712 29610 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:30:28.427943 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 29531
W20260618 23:30:28.432204 29516 connection.cc:535] client connection to 127.8.39.190:42975 recv error: Network error: failed to read from TLS socket (remote: 127.8.39.190:42975): Connection reset by peer (error 104)
2026-06-18T23:30:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1332 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:30:28 dist-test-slave-47qt krb5kdc[29630](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:30:28 dist-test-slave-47qt krb5kdc[29630](info): set up 2 sockets
Jun 18 23:30:28 dist-test-slave-47qt krb5kdc[29630](info): commencing operation
krb5kdc: starting...
W20260618 23:30:29.493201 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:30:29 dist-test-slave-47qt krb5kdc[29630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825429, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:30:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:30:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.39.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
I20260618 23:30:29.612434 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:36317
--webserver_interface=127.8.39.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:34957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:36317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:30:29.691133 29643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:29.691380 29643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:29.691416 29643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:29.692536 29643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:30:29.692608 29643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:29.692636 29643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:29.692667 29643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:30:29.692690 29643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:30:29.694172 29643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:34957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:36317
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:36317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:29.694439 29643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:29.694658 29643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 23:30:29.699611 29643 server_base.cc:1034] running on GCE node
W20260618 23:30:29.699517 29649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:29.699544 29648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:29.699510 29651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:29.700047 29643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:29.700363 29643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:29.701573 29643 hybrid_clock.cc:648] HybridClock initialized: now 1781825429701553 us; error 31 us; skew 500 ppm
Jun 18 23:30:29 dist-test-slave-47qt krb5kdc[29630](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825429, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:29.704070 29643 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:30:29.704527 29643 webserver.cc:460] Webserver started at http://127.8.39.190:36291/ using document root <none> and password file <none>
I20260618 23:30:29.704706 29643 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:29.704747 29643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:29.704833 29643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:29.705874 29643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "a4410038f0a046ddb85dc7a7c976d9f0"
format_stamp: "Formatted at 2026-06-18 23:30:29 on dist-test-slave-47qt"
server_key: "e28ca7b93bf59527ad0ecb0ab9c1c21f"
server_key_iv: "e331e1b0736e76309d3833803ed4e5b9"
server_key_version: "encryptionkey@0"
I20260618 23:30:29.706171 29643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "a4410038f0a046ddb85dc7a7c976d9f0"
format_stamp: "Formatted at 2026-06-18 23:30:29 on dist-test-slave-47qt"
server_key: "e28ca7b93bf59527ad0ecb0ab9c1c21f"
server_key_iv: "e331e1b0736e76309d3833803ed4e5b9"
server_key_version: "encryptionkey@0"
I20260618 23:30:29.707376 29643 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:30:29.708029 29658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:29.708252 29643 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260618 23:30:29.708343 29643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "a4410038f0a046ddb85dc7a7c976d9f0"
format_stamp: "Formatted at 2026-06-18 23:30:29 on dist-test-slave-47qt"
server_key: "e28ca7b93bf59527ad0ecb0ab9c1c21f"
server_key_iv: "e331e1b0736e76309d3833803ed4e5b9"
server_key_version: "encryptionkey@0"
I20260618 23:30:29.708413 29643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:29.711627 29643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:29.714210 29643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:29.714365 29643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:29.718744 29643 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:36317
I20260618 23:30:29.718794 29710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:36317 every 8 connection(s)
I20260618 23:30:29.719086 29643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:30:29.720293 29711 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:29.722612 29711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4410038f0a046ddb85dc7a7c976d9f0: Bootstrap starting.
I20260618 23:30:29.723177 29711 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4410038f0a046ddb85dc7a7c976d9f0: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:29.723453 29711 log.cc:824] T 00000000000000000000000000000000 P a4410038f0a046ddb85dc7a7c976d9f0: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:29.724113 29711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4410038f0a046ddb85dc7a7c976d9f0: No bootstrap required, opened a new log
I20260618 23:30:29.725718 29711 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4410038f0a046ddb85dc7a7c976d9f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4410038f0a046ddb85dc7a7c976d9f0" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 36317 } }
I20260618 23:30:29.725849 29711 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4410038f0a046ddb85dc7a7c976d9f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:29.725872 29711 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4410038f0a046ddb85dc7a7c976d9f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4410038f0a046ddb85dc7a7c976d9f0, State: Initialized, Role: FOLLOWER
I20260618 23:30:29.726022 29711 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4410038f0a046ddb85dc7a7c976d9f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4410038f0a046ddb85dc7a7c976d9f0" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 36317 } }
I20260618 23:30:29.726083 29711 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4410038f0a046ddb85dc7a7c976d9f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:30:29.726104 29711 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4410038f0a046ddb85dc7a7c976d9f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:30:29.726159 29711 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4410038f0a046ddb85dc7a7c976d9f0 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:29.726935 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 29643
I20260618 23:30:29.726958 29711 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4410038f0a046ddb85dc7a7c976d9f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4410038f0a046ddb85dc7a7c976d9f0" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 36317 } }
I20260618 23:30:29.727025 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:30:29.727072 29711 leader_election.cc:302] T 00000000000000000000000000000000 P a4410038f0a046ddb85dc7a7c976d9f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4410038f0a046ddb85dc7a7c976d9f0; no voters:
I20260618 23:30:29.727262 8350 external_mini_cluster.cc:1428] Setting key c8a68d9311dfbf0d8724e12093ebe835
I20260618 23:30:29.727272 29711 leader_election.cc:288] T 00000000000000000000000000000000 P a4410038f0a046ddb85dc7a7c976d9f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:30:29.727299 29714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4410038f0a046ddb85dc7a7c976d9f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:30:29.727406 29714 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4410038f0a046ddb85dc7a7c976d9f0 [term 1 LEADER]: Becoming Leader. State: Replica: a4410038f0a046ddb85dc7a7c976d9f0, State: Running, Role: LEADER
I20260618 23:30:29.727526 29711 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4410038f0a046ddb85dc7a7c976d9f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:30:29.727558 29714 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4410038f0a046ddb85dc7a7c976d9f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4410038f0a046ddb85dc7a7c976d9f0" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 36317 } }
I20260618 23:30:29.727941 29715 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4410038f0a046ddb85dc7a7c976d9f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4410038f0a046ddb85dc7a7c976d9f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4410038f0a046ddb85dc7a7c976d9f0" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 36317 } } }
I20260618 23:30:29.728037 29715 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4410038f0a046ddb85dc7a7c976d9f0 [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:29.728063 29716 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4410038f0a046ddb85dc7a7c976d9f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4410038f0a046ddb85dc7a7c976d9f0. Latest consensus state: current_term: 1 leader_uuid: "a4410038f0a046ddb85dc7a7c976d9f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4410038f0a046ddb85dc7a7c976d9f0" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 36317 } } }
I20260618 23:30:29.728139 29716 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4410038f0a046ddb85dc7a7c976d9f0 [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:29.728355 29724 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:30:29.729147 29724 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:30:29.729672 29719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:29.727833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33973 (local address 127.8.39.190:36317)
0618 23:30:29.728007 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:29.728015 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:29.728043 (+ 28us) server_negotiation.cc:407] Connection header received
0618 23:30:29.728168 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:29.728175 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:29.728207 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:29.728598 (+ 391us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:29.729077 (+ 479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:29.729080 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:29.729081 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:29.729084 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:29.729158 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:29.729172 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:29.729434 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:29.729464 (+ 30us) server_negotiation.cc:299] Negotiation successful
0618 23:30:29.729501 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":73,"threads_started":1}
I20260618 23:30:29.730731 29724 catalog_manager.cc:1347] Generated new cluster ID: 259913bffd194f369fbe5e908a90826e
I20260618 23:30:29.730782 29724 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:30:29.730796 8350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:30:29.733341 29719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:29.732286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42435 (local address 127.8.39.190:36317)
0618 23:30:29.732325 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:29.732327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:29.732464 (+ 137us) server_negotiation.cc:407] Connection header received
0618 23:30:29.732512 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:29.732513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:29.732518 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:29.732577 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:29.733152 (+ 575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:29.733153 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:29.733153 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:29.733156 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:29.733170 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:29.733173 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:29.733227 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:29.733275 (+ 48us) server_negotiation.cc:299] Negotiation successful
0618 23:30:29.733286 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260618 23:30:29.744549 29724 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:30:29.744714 29724 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:30:29.753559 29724 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4410038f0a046ddb85dc7a7c976d9f0: Generated new TSK 0
I20260618 23:30:29.753861 29724 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:30:29.758719 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 29643
2026-06-18T23:30:29Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1330 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:30:29 dist-test-slave-47qt krb5kdc[29742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:30:29 dist-test-slave-47qt krb5kdc[29742](info): set up 2 sockets
Jun 18 23:30:29 dist-test-slave-47qt krb5kdc[29742](info): commencing operation
krb5kdc: starting...
W20260618 23:30:30.829367 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.053s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:30:30 dist-test-slave-47qt krb5kdc[29742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:30:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:30:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.39.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
I20260618 23:30:30.948688 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:43539
--webserver_interface=127.8.39.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:37437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:43539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:30:31.030846 29755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:31.031095 29755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:31.031134 29755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:31.032305 29755 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:30:31.032377 29755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:31.032403 29755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:31.032436 29755 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:30:31.032460 29755 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:30:31.034052 29755 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:37437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:43539
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:43539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:31.034338 29755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:31.034569 29755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:31.039745 29760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:31.039745 29761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:31.040215 29763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:31.039922 29755 server_base.cc:1034] running on GCE node
I20260618 23:30:31.040654 29755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:31.041224 29755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:31.042389 29755 hybrid_clock.cc:648] HybridClock initialized: now 1781825431042354 us; error 52 us; skew 500 ppm
Jun 18 23:30:31 dist-test-slave-47qt krb5kdc[29742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825431, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:31.044957 29755 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:30:31.045423 29755 webserver.cc:460] Webserver started at http://127.8.39.190:37471/ using document root <none> and password file <none>
I20260618 23:30:31.045665 29755 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:31.045709 29755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:31.045853 29755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:31.046896 29755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "ae44ce5011364759889f7ac78c2a014b"
format_stamp: "Formatted at 2026-06-18 23:30:31 on dist-test-slave-47qt"
server_key: "11f4ed49eb84ca12d186d21790986b1a"
server_key_iv: "606c8c08d02d6d32937844277d29e406"
server_key_version: "encryptionkey@0"
I20260618 23:30:31.047266 29755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "ae44ce5011364759889f7ac78c2a014b"
format_stamp: "Formatted at 2026-06-18 23:30:31 on dist-test-slave-47qt"
server_key: "11f4ed49eb84ca12d186d21790986b1a"
server_key_iv: "606c8c08d02d6d32937844277d29e406"
server_key_version: "encryptionkey@0"
I20260618 23:30:31.048651 29755 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260618 23:30:31.049602 29770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:31.049850 29755 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260618 23:30:31.049952 29755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "ae44ce5011364759889f7ac78c2a014b"
format_stamp: "Formatted at 2026-06-18 23:30:31 on dist-test-slave-47qt"
server_key: "11f4ed49eb84ca12d186d21790986b1a"
server_key_iv: "606c8c08d02d6d32937844277d29e406"
server_key_version: "encryptionkey@0"
I20260618 23:30:31.050027 29755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:31.078657 29755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:31.080866 29755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:31.081027 29755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:31.086303 29755 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:43539
I20260618 23:30:31.086338 29822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:43539 every 8 connection(s)
I20260618 23:30:31.086684 29755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:30:31.087564 29823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:31.089870 29823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae44ce5011364759889f7ac78c2a014b: Bootstrap starting.
I20260618 23:30:31.090473 29823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae44ce5011364759889f7ac78c2a014b: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:31.090768 29823 log.cc:824] T 00000000000000000000000000000000 P ae44ce5011364759889f7ac78c2a014b: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:31.091528 29823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae44ce5011364759889f7ac78c2a014b: No bootstrap required, opened a new log
I20260618 23:30:31.093168 29823 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae44ce5011364759889f7ac78c2a014b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae44ce5011364759889f7ac78c2a014b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 43539 } }
I20260618 23:30:31.093307 29823 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae44ce5011364759889f7ac78c2a014b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:31.093328 29823 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae44ce5011364759889f7ac78c2a014b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae44ce5011364759889f7ac78c2a014b, State: Initialized, Role: FOLLOWER
I20260618 23:30:31.093488 29823 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae44ce5011364759889f7ac78c2a014b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae44ce5011364759889f7ac78c2a014b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 43539 } }
I20260618 23:30:31.093552 29823 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae44ce5011364759889f7ac78c2a014b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:30:31.093588 29823 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae44ce5011364759889f7ac78c2a014b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:30:31.093645 29823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae44ce5011364759889f7ac78c2a014b [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:31.093766 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 29755
I20260618 23:30:31.093899 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:30:31.094074 8350 external_mini_cluster.cc:1428] Setting key 3bdec763c1aee038fbacf83dbab24130
I20260618 23:30:31.094771 29823 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae44ce5011364759889f7ac78c2a014b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae44ce5011364759889f7ac78c2a014b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 43539 } }
I20260618 23:30:31.094905 29823 leader_election.cc:302] T 00000000000000000000000000000000 P ae44ce5011364759889f7ac78c2a014b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae44ce5011364759889f7ac78c2a014b; no voters:
I20260618 23:30:31.095058 29823 leader_election.cc:288] T 00000000000000000000000000000000 P ae44ce5011364759889f7ac78c2a014b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:30:31.095083 29828 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae44ce5011364759889f7ac78c2a014b [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:30:31.095279 29828 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae44ce5011364759889f7ac78c2a014b [term 1 LEADER]: Becoming Leader. State: Replica: ae44ce5011364759889f7ac78c2a014b, State: Running, Role: LEADER
I20260618 23:30:31.095394 29823 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae44ce5011364759889f7ac78c2a014b [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:30:31.095402 29828 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae44ce5011364759889f7ac78c2a014b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae44ce5011364759889f7ac78c2a014b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 43539 } }
I20260618 23:30:31.095853 29829 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae44ce5011364759889f7ac78c2a014b [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae44ce5011364759889f7ac78c2a014b. Latest consensus state: current_term: 1 leader_uuid: "ae44ce5011364759889f7ac78c2a014b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae44ce5011364759889f7ac78c2a014b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 43539 } } }
I20260618 23:30:31.096005 29829 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae44ce5011364759889f7ac78c2a014b [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:31.096187 29830 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae44ce5011364759889f7ac78c2a014b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae44ce5011364759889f7ac78c2a014b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae44ce5011364759889f7ac78c2a014b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 43539 } } }
I20260618 23:30:31.096292 29830 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae44ce5011364759889f7ac78c2a014b [sys.catalog]: This master's current role is: LEADER
W20260618 23:30:31.097206 29844 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ae44ce5011364759889f7ac78c2a014b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 23:30:31.097265 29844 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:30:31.097308 29837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:30:31.097448 29837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:30:31.098973 29837 catalog_manager.cc:1347] Generated new cluster ID: a814e9aecba843908e6f58c311926516
I20260618 23:30:31.099030 29837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 18 23:30:31 dist-test-slave-47qt krb5kdc[29742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825430, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:30:31.107343 29827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:31.094598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58191 (local address 127.8.39.190:43539)
0618 23:30:31.094782 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:31.094797 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:31.094822 (+ 25us) server_negotiation.cc:407] Connection header received
0618 23:30:31.094938 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:31.094945 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:31.094999 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:31.095160 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:31.095903 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:31.097407 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:31.098264 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:31.098851 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:31.101451 (+ 2600us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:31.104047 (+ 2596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:31.104052 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:31.104052 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:31.104064 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:31.105805 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:31.106174 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:31.106177 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:31.106178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:31.106280 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:31.106487 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:31.106488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:31.106488 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:31.106736 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:31.106910 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:31.106982 (+ 72us) server_negotiation.cc:299] Negotiation successful
0618 23:30:31.107122 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":112,"threads_started":1}
I20260618 23:30:31.108500 8350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:30:31.109380 29837 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:30:31.110101 29837 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:30:31.114058 29837 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae44ce5011364759889f7ac78c2a014b: Generated new TSK 0
I20260618 23:30:31.114254 29837 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:30:31.120481 29827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:31.109910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56237 (local address 127.8.39.190:43539)
0618 23:30:31.109947 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:31.109949 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:31.113462 (+ 3513us) server_negotiation.cc:407] Connection header received
0618 23:30:31.113748 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:31.113748 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:31.113889 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:31.113977 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:31.114640 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:31.115104 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:31.116184 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:31.116449 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:31.116540 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:31.117953 (+ 1413us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:31.117955 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:31.117955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:31.117964 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:31.119384 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:31.119791 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:31.119792 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:31.119792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:31.119846 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:31.120048 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:31.120049 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:31.120049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:31.120158 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:31.120234 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:31.120361 (+ 127us) server_negotiation.cc:299] Negotiation successful
0618 23:30:31.120424 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260618 23:30:31.123169 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 29755
2026-06-18T23:30:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1364 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:30:31 dist-test-slave-47qt krb5kdc[29854](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:30:31 dist-test-slave-47qt krb5kdc[29854](info): set up 2 sockets
Jun 18 23:30:31 dist-test-slave-47qt krb5kdc[29854](info): commencing operation
krb5kdc: starting...
W20260618 23:30:32.169278 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:30:32 dist-test-slave-47qt krb5kdc[29854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825432, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:30:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:30:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.39.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
I20260618 23:30:32.291941 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:42735
--webserver_interface=127.8.39.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:37451
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:42735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:30:32.376839 29867 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:32.377058 29867 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:32.377092 29867 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:32.378336 29867 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:30:32.378381 29867 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:32.378397 29867 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:32.378409 29867 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:30:32.378422 29867 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:30:32.379918 29867 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:37451
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:42735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:42735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29867
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:32.380185 29867 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:32.380415 29867 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 23:30:32.385531 29867 server_base.cc:1034] running on GCE node
W20260618 23:30:32.385505 29875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:32.385478 29873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:32.385505 29872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:32.386025 29867 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:32.386282 29867 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:32.387471 29867 hybrid_clock.cc:648] HybridClock initialized: now 1781825432387458 us; error 31 us; skew 500 ppm
Jun 18 23:30:32 dist-test-slave-47qt krb5kdc[29854](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825432, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:32.393780 29867 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:30:32.394296 29867 webserver.cc:460] Webserver started at http://127.8.39.190:33641/ using document root <none> and password file <none>
I20260618 23:30:32.394559 29867 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:32.394606 29867 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:32.394745 29867 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:32.396284 29867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "c1e8bc276ac8454bbfbb68da5f35dba6"
format_stamp: "Formatted at 2026-06-18 23:30:32 on dist-test-slave-47qt"
server_key: "a19ee30556f21e640d096c98e7a794e0"
server_key_iv: "1dfc040ecac3bca9706bd37a09560add"
server_key_version: "encryptionkey@0"
I20260618 23:30:32.396798 29867 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "c1e8bc276ac8454bbfbb68da5f35dba6"
format_stamp: "Formatted at 2026-06-18 23:30:32 on dist-test-slave-47qt"
server_key: "a19ee30556f21e640d096c98e7a794e0"
server_key_iv: "1dfc040ecac3bca9706bd37a09560add"
server_key_version: "encryptionkey@0"
I20260618 23:30:32.398729 29867 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260618 23:30:32.399981 29882 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:32.400259 29867 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:30:32.400357 29867 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "c1e8bc276ac8454bbfbb68da5f35dba6"
format_stamp: "Formatted at 2026-06-18 23:30:32 on dist-test-slave-47qt"
server_key: "a19ee30556f21e640d096c98e7a794e0"
server_key_iv: "1dfc040ecac3bca9706bd37a09560add"
server_key_version: "encryptionkey@0"
I20260618 23:30:32.400444 29867 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:32.424341 29867 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:32.426860 29867 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:32.426980 29867 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:32.430980 29867 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:42735
I20260618 23:30:32.431034 29934 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:42735 every 8 connection(s)
I20260618 23:30:32.431378 29867 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:30:32.432303 29935 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:32.434536 29935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1e8bc276ac8454bbfbb68da5f35dba6: Bootstrap starting.
I20260618 23:30:32.435119 29935 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1e8bc276ac8454bbfbb68da5f35dba6: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:32.435408 29935 log.cc:824] T 00000000000000000000000000000000 P c1e8bc276ac8454bbfbb68da5f35dba6: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:32.436462 29935 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1e8bc276ac8454bbfbb68da5f35dba6: No bootstrap required, opened a new log
I20260618 23:30:32.437685 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 29867
I20260618 23:30:32.437830 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:30:32.438020 8350 external_mini_cluster.cc:1428] Setting key 8bb4c92f7cd8344e272346b2cd8dbeca
I20260618 23:30:32.439956 29935 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1e8bc276ac8454bbfbb68da5f35dba6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e8bc276ac8454bbfbb68da5f35dba6" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 42735 } }
I20260618 23:30:32.440151 29935 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1e8bc276ac8454bbfbb68da5f35dba6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:32.440199 29935 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1e8bc276ac8454bbfbb68da5f35dba6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1e8bc276ac8454bbfbb68da5f35dba6, State: Initialized, Role: FOLLOWER
I20260618 23:30:32.440369 29935 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1e8bc276ac8454bbfbb68da5f35dba6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e8bc276ac8454bbfbb68da5f35dba6" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 42735 } }
I20260618 23:30:32.440460 29935 raft_consensus.cc:388] T 00000000000000000000000000000000 P c1e8bc276ac8454bbfbb68da5f35dba6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:30:32.440492 29935 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1e8bc276ac8454bbfbb68da5f35dba6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:30:32.440531 29935 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1e8bc276ac8454bbfbb68da5f35dba6 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:32.441313 29935 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1e8bc276ac8454bbfbb68da5f35dba6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e8bc276ac8454bbfbb68da5f35dba6" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 42735 } }
I20260618 23:30:32.441419 29935 leader_election.cc:302] T 00000000000000000000000000000000 P c1e8bc276ac8454bbfbb68da5f35dba6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1e8bc276ac8454bbfbb68da5f35dba6; no voters:
I20260618 23:30:32.441555 29935 leader_election.cc:288] T 00000000000000000000000000000000 P c1e8bc276ac8454bbfbb68da5f35dba6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:30:32.441622 29940 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1e8bc276ac8454bbfbb68da5f35dba6 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:30:32.441804 29940 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1e8bc276ac8454bbfbb68da5f35dba6 [term 1 LEADER]: Becoming Leader. State: Replica: c1e8bc276ac8454bbfbb68da5f35dba6, State: Running, Role: LEADER
I20260618 23:30:32.441927 29940 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1e8bc276ac8454bbfbb68da5f35dba6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e8bc276ac8454bbfbb68da5f35dba6" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 42735 } }
I20260618 23:30:32.441815 29935 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1e8bc276ac8454bbfbb68da5f35dba6 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:30:32.442644 29942 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1e8bc276ac8454bbfbb68da5f35dba6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1e8bc276ac8454bbfbb68da5f35dba6. Latest consensus state: current_term: 1 leader_uuid: "c1e8bc276ac8454bbfbb68da5f35dba6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e8bc276ac8454bbfbb68da5f35dba6" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 42735 } } }
I20260618 23:30:32.442796 29942 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1e8bc276ac8454bbfbb68da5f35dba6 [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:32.443216 29941 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1e8bc276ac8454bbfbb68da5f35dba6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1e8bc276ac8454bbfbb68da5f35dba6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1e8bc276ac8454bbfbb68da5f35dba6" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 42735 } } }
I20260618 23:30:32.443511 29941 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1e8bc276ac8454bbfbb68da5f35dba6 [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:32.443683 29948 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:30:32.444306 29948 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:30:32.445147 29939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:32.438596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41891 (local address 127.8.39.190:42735)
0618 23:30:32.438841 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:32.438849 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:32.438873 (+ 24us) server_negotiation.cc:407] Connection header received
0618 23:30:32.439861 (+ 988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:32.439867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:32.439915 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:32.440074 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:32.440296 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:32.441696 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:32.442872 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:32.443519 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:32.443997 (+ 478us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:32.444426 (+ 429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:32.444430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:32.444430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:32.444607 (+ 177us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:32.444636 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:32.444646 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:32.444695 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:32.444843 (+ 148us) server_negotiation.cc:299] Negotiation successful
0618 23:30:32.444909 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":100,"threads_started":1}
I20260618 23:30:32.445974 29948 catalog_manager.cc:1347] Generated new cluster ID: b77a5905e29d4934b10966a1dd60c819
I20260618 23:30:32.446024 29948 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:30:32.446063 8350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:30:32.452662 29939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:32.447559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38513 (local address 127.8.39.190:42735)
0618 23:30:32.447595 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:32.447596 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:32.449330 (+ 1734us) server_negotiation.cc:407] Connection header received
0618 23:30:32.449434 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:32.449435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:32.449448 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:32.449517 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:32.449664 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:32.450360 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:32.451384 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:32.451657 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:32.451716 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:32.452439 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:32.452440 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:32.452440 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:32.452444 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:32.452459 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:32.452463 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:32.452510 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:32.452564 (+ 54us) server_negotiation.cc:299] Negotiation successful
0618 23:30:32.452585 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260618 23:30:32.456434 29948 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:30:32.457163 29948 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:30:32.460064 29948 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1e8bc276ac8454bbfbb68da5f35dba6: Generated new TSK 0
I20260618 23:30:32.460218 29948 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:30:32.462806 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 29867
2026-06-18T23:30:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1339 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:30:32 dist-test-slave-47qt krb5kdc[29966](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:30:32 dist-test-slave-47qt krb5kdc[29966](info): set up 2 sockets
Jun 18 23:30:32 dist-test-slave-47qt krb5kdc[29966](info): commencing operation
krb5kdc: starting...
W20260618 23:30:33.517349 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:30:33 dist-test-slave-47qt krb5kdc[29966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825433, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:30:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:30:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.39.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
I20260618 23:30:33.649214 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:33207
--webserver_interface=127.8.39.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:37067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:33207
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:30:33.732357 29979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:33.732622 29979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:33.732661 29979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:33.733853 29979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:30:33.733924 29979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:33.733951 29979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:33.733983 29979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:30:33.734004 29979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:30:33.735427 29979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:37067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:33207
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:33207
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:33.735702 29979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:33.735924 29979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:33.740612 29985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:33.740654 29984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:33.740813 29979 server_base.cc:1034] running on GCE node
W20260618 23:30:33.740931 29987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:33.741351 29979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:33.741660 29979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:33.742825 29979 hybrid_clock.cc:648] HybridClock initialized: now 1781825433742791 us; error 49 us; skew 500 ppm
Jun 18 23:30:33 dist-test-slave-47qt krb5kdc[29966](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825433, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:33.745682 29979 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:30:33.746326 29979 webserver.cc:460] Webserver started at http://127.8.39.190:39487/ using document root <none> and password file <none>
I20260618 23:30:33.746590 29979 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:33.746662 29979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:33.746816 29979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:33.747874 29979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "1b70ed6ad23d42619e65e4ac6da19a87"
format_stamp: "Formatted at 2026-06-18 23:30:33 on dist-test-slave-47qt"
server_key: "e8b4bb8380900e740107757455d73313"
server_key_iv: "e03b8201afb7f5cdf9a5aae5ab0d52f4"
server_key_version: "encryptionkey@0"
I20260618 23:30:33.748242 29979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "1b70ed6ad23d42619e65e4ac6da19a87"
format_stamp: "Formatted at 2026-06-18 23:30:33 on dist-test-slave-47qt"
server_key: "e8b4bb8380900e740107757455d73313"
server_key_iv: "e03b8201afb7f5cdf9a5aae5ab0d52f4"
server_key_version: "encryptionkey@0"
I20260618 23:30:33.749631 29979 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:30:33.751026 29994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:33.751292 29979 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:30:33.751392 29979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "1b70ed6ad23d42619e65e4ac6da19a87"
format_stamp: "Formatted at 2026-06-18 23:30:33 on dist-test-slave-47qt"
server_key: "e8b4bb8380900e740107757455d73313"
server_key_iv: "e03b8201afb7f5cdf9a5aae5ab0d52f4"
server_key_version: "encryptionkey@0"
I20260618 23:30:33.751479 29979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:33.754907 29979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:33.757241 29979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:33.757400 29979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:33.761371 29979 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:33207
I20260618 23:30:33.761420 30046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:33207 every 8 connection(s)
I20260618 23:30:33.761761 29979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:30:33.762732 30047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:33.764338 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 29979
I20260618 23:30:33.764530 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:30:33.764781 8350 external_mini_cluster.cc:1428] Setting key c29e91a9aaba245e2b2d5f5e7ffd1939
I20260618 23:30:33.765161 30047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b70ed6ad23d42619e65e4ac6da19a87: Bootstrap starting.
I20260618 23:30:33.765882 30047 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b70ed6ad23d42619e65e4ac6da19a87: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:33.766170 30047 log.cc:824] T 00000000000000000000000000000000 P 1b70ed6ad23d42619e65e4ac6da19a87: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:33.766954 30047 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b70ed6ad23d42619e65e4ac6da19a87: No bootstrap required, opened a new log
I20260618 23:30:33.767648 30050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:33.765524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54597 (local address 127.8.39.190:33207)
0618 23:30:33.765756 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:33.765763 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:33.765788 (+ 25us) server_negotiation.cc:407] Connection header received
0618 23:30:33.766328 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:33.766335 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:33.766390 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:33.766544 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:33.767228 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:33.767232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:33.767232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:33.767236 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:33.767284 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:33.767294 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:33.767406 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:33.767446 (+ 40us) server_negotiation.cc:299] Negotiation successful
0618 23:30:33.767484 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":109,"threads_started":1}
I20260618 23:30:33.768752 8350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:30:33.769296 30047 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b70ed6ad23d42619e65e4ac6da19a87 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b70ed6ad23d42619e65e4ac6da19a87" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33207 } }
I20260618 23:30:33.769448 30047 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b70ed6ad23d42619e65e4ac6da19a87 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260618 23:30:33.769492 30047 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b70ed6ad23d42619e65e4ac6da19a87 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b70ed6ad23d42619e65e4ac6da19a87, State: Initialized, Role: FOLLOWER
I20260618 23:30:33.769631 30047 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b70ed6ad23d42619e65e4ac6da19a87 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b70ed6ad23d42619e65e4ac6da19a87" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33207 } }
I20260618 23:30:33.769718 30047 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b70ed6ad23d42619e65e4ac6da19a87 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260618 23:30:33.769753 30047 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b70ed6ad23d42619e65e4ac6da19a87 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260618 23:30:33.769800 30047 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b70ed6ad23d42619e65e4ac6da19a87 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:33.770515 30047 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b70ed6ad23d42619e65e4ac6da19a87 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b70ed6ad23d42619e65e4ac6da19a87" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33207 } }
I20260618 23:30:33.770605 30047 leader_election.cc:302] T 00000000000000000000000000000000 P 1b70ed6ad23d42619e65e4ac6da19a87 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b70ed6ad23d42619e65e4ac6da19a87; no voters:
I20260618 23:30:33.770780 30047 leader_election.cc:288] T 00000000000000000000000000000000 P 1b70ed6ad23d42619e65e4ac6da19a87 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260618 23:30:33.770838 30057 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b70ed6ad23d42619e65e4ac6da19a87 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:30:33.771023 30047 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b70ed6ad23d42619e65e4ac6da19a87 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:30:33.771231 30057 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b70ed6ad23d42619e65e4ac6da19a87 [term 1 LEADER]: Becoming Leader. State: Replica: 1b70ed6ad23d42619e65e4ac6da19a87, State: Running, Role: LEADER
I20260618 23:30:33.771332 30050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:33.770259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36995 (local address 127.8.39.190:33207)
0618 23:30:33.770297 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:33.770300 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:33.770345 (+ 45us) server_negotiation.cc:407] Connection header received
0618 23:30:33.770418 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:33.770419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:33.770424 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:33.770479 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:33.771038 (+ 559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:33.771039 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:33.771039 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0618 23:30:33.771043 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:33.771058 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0618 23:30:33.771061 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:33.771108 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:33.771214 (+ 106us) server_negotiation.cc:299] Negotiation successful
0618 23:30:33.771230 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260618 23:30:33.771399 30057 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b70ed6ad23d42619e65e4ac6da19a87 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b70ed6ad23d42619e65e4ac6da19a87" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33207 } }
I20260618 23:30:33.771899 30059 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b70ed6ad23d42619e65e4ac6da19a87 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b70ed6ad23d42619e65e4ac6da19a87. Latest consensus state: current_term: 1 leader_uuid: "1b70ed6ad23d42619e65e4ac6da19a87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b70ed6ad23d42619e65e4ac6da19a87" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33207 } } }
I20260618 23:30:33.771991 30059 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b70ed6ad23d42619e65e4ac6da19a87 [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:33.772114 30058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b70ed6ad23d42619e65e4ac6da19a87 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b70ed6ad23d42619e65e4ac6da19a87" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b70ed6ad23d42619e65e4ac6da19a87" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 33207 } } }
I20260618 23:30:33.772195 30058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b70ed6ad23d42619e65e4ac6da19a87 [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:33.772226 30066 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:30:33.772686 30066 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:30:33.774269 30066 catalog_manager.cc:1347] Generated new cluster ID: 1ad5b475288f4b7f83df6f3615b2227f
I20260618 23:30:33.774322 30066 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:30:33.789605 30066 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:30:33.789867 30066 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:30:33.795311 30066 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b70ed6ad23d42619e65e4ac6da19a87: Generated new TSK 0
I20260618 23:30:33.795516 30066 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:30:33.802325 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 29979
2026-06-18T23:30:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1340 ms)
[----------] 18 tests from AuthTokenIssuingTest (24445 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:30:33 dist-test-slave-47qt krb5kdc[30078](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:30:33 dist-test-slave-47qt krb5kdc[30078](info): set up 2 sockets
Jun 18 23:30:33 dist-test-slave-47qt krb5kdc[30078](info): commencing operation
krb5kdc: starting...
W20260618 23:30:34.893735 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:30:34 dist-test-slave-47qt krb5kdc[30078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:30:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:30:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.39.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
I20260618 23:30:35.016532 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:32995
--webserver_interface=127.8.39.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:35223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:32995,127.8.39.189:44545,127.8.39.188:36161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:30:35.099628 30091 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260618 23:30:35.099855 30091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:35.099874 30091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:35.099893 30091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:35.101052 30091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:30:35.101094 30091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:35.101109 30091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:35.101145 30091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:30:35.101157 30091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:30:35.102640 30091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:35223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:32995,127.8.39.189:44545,127.8.39.188:36161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:32995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30091
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:35.102905 30091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:35.103135 30091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:35.108260 30099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:35.108202 30096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:35.108207 30097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:35.108572 30091 server_base.cc:1034] running on GCE node
I20260618 23:30:35.108844 30091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:35.109251 30091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:35.110445 30091 hybrid_clock.cc:648] HybridClock initialized: now 1781825435110432 us; error 32 us; skew 500 ppm
Jun 18 23:30:35 dist-test-slave-47qt krb5kdc[30078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825435, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:35.113061 30091 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:30:35.113555 30091 webserver.cc:460] Webserver started at http://127.8.39.190:33791/ using document root <none> and password file <none>
I20260618 23:30:35.113780 30091 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:35.113863 30091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:35.113996 30091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:35.115041 30091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "fab5f31acc1a4190b59546b902a58998"
format_stamp: "Formatted at 2026-06-18 23:30:35 on dist-test-slave-47qt"
server_key: "d122561ad9be16a15d0c89b4289f0795"
server_key_iv: "7b46bbc287f51e325188005aeb8fc552"
server_key_version: "encryptionkey@0"
I20260618 23:30:35.115398 30091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "fab5f31acc1a4190b59546b902a58998"
format_stamp: "Formatted at 2026-06-18 23:30:35 on dist-test-slave-47qt"
server_key: "d122561ad9be16a15d0c89b4289f0795"
server_key_iv: "7b46bbc287f51e325188005aeb8fc552"
server_key_version: "encryptionkey@0"
I20260618 23:30:35.116708 30091 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:30:35.117570 30106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:35.117806 30091 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260618 23:30:35.117918 30091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "fab5f31acc1a4190b59546b902a58998"
format_stamp: "Formatted at 2026-06-18 23:30:35 on dist-test-slave-47qt"
server_key: "d122561ad9be16a15d0c89b4289f0795"
server_key_iv: "7b46bbc287f51e325188005aeb8fc552"
server_key_version: "encryptionkey@0"
I20260618 23:30:35.118009 30091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:35.131844 30091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:35.133745 30091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:35.133908 30091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:35.137919 30091 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:32995
I20260618 23:30:35.137951 30158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:32995 every 8 connection(s)
I20260618 23:30:35.138294 30091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:30:35.139241 30159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:35.141312 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 30091
I20260618 23:30:35.141420 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:30:35.141613 8350 external_mini_cluster.cc:1428] Setting key fb087c30f3943c8b7726a39e02b52dbf
I20260618 23:30:35.141454 30159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.39.190" port: 32995 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 23:30:35 dist-test-slave-47qt krb5kdc[30078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825435, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.39.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.189@KRBTEST.COM" created.
I20260618 23:30:35.152395 30161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.142277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:32995 (local address 127.0.0.1:50226)
0618 23:30:35.142466 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:35.142507 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:35.143072 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:35.143367 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:35.143369 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:35.143653 (+ 284us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:35.143797 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:35.143800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.145003 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.145004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:35.145786 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:35.145788 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.146317 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.146318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:35.146410 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:35.146969 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:30:35.146977 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:35.148530 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:35.150373 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:35.150379 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:35.150380 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:35.150573 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:35.150840 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:35.150841 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:35.150841 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:35.150949 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:35.151384 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:35.151399 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:35.151575 (+ 176us) client_negotiation.cc:769] Sending connection context
0618 23:30:35.151788 (+ 213us) client_negotiation.cc:240] Negotiation successful
0618 23:30:35.152247 (+ 459us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":135,"thread_start_us":105,"threads_started":1}
I20260618 23:30:35.152405 30162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.142435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50226 (local address 127.8.39.190:32995)
0618 23:30:35.142683 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:35.142687 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:35.142717 (+ 30us) server_negotiation.cc:407] Connection header received
0618 23:30:35.143226 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:35.143228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:35.143260 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:35.143362 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:35.143894 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.144808 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.145902 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.146202 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.146331 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:35.148625 (+ 2294us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:35.148634 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:35.148635 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:35.148643 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:35.150301 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:35.150670 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:35.150672 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:35.150673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:35.150768 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:35.151030 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:35.151031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:35.151031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:35.151248 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:35.151322 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:35.151834 (+ 512us) server_negotiation.cc:299] Negotiation successful
0618 23:30:35.152119 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":176,"threads_started":1}
I20260618 23:30:35.153614 30159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44545 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:30:35.153975 30161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.153850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:44545 (local address 127.0.0.1:54560)
0618 23:30:35.153903 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:35.153945 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.189:44545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260618 23:30:35.154148 30108 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.39.189:44545: connect: Connection refused (error 111)
I20260618 23:30:35.154745 30161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.154523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:44545 (local address 127.0.0.1:54576)
0618 23:30:35.154626 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:35.154686 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.189:44545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260618 23:30:35.154964 30159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.39.189:44545: Network error: Client connection negotiation failed: client connection to 127.8.39.189:44545: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.189.keytab.
Entry for principal kudu/127.8.39.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.189.keytab.
I20260618 23:30:35.166352 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.189:44545
--webserver_interface=127.8.39.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.189.keytab
--principal=kudu/127.8.39.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:35223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:32995,127.8.39.189:44545,127.8.39.188:36161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:30:35.204254 30159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44545 } attempt: 1
I20260618 23:30:35.204768 30161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.204618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:44545 (local address 127.0.0.1:54580)
0618 23:30:35.204692 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:35.204740 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.189:44545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260618 23:30:35.205098 30161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.205006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:44545 (local address 127.0.0.1:54582)
0618 23:30:35.205050 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:35.205074 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.189:44545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260618 23:30:35.205199 30159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.39.189:44545: Network error: Client connection negotiation failed: client connection to 127.8.39.189:44545: connect: Connection refused (error 111)
W20260618 23:30:35.249497 30166 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260618 23:30:35.249727 30166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:35.249744 30166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:35.249766 30166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:35.251009 30166 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:30:35.251053 30166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:35.251068 30166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:35.251081 30166 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:30:35.251139 30166 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:30:35.252781 30166 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:35223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:32995,127.8.39.189:44545,127.8.39.188:36161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.189.keytab
--principal=kudu/127.8.39.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.189:44545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.39.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:35.253059 30166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:35.253336 30166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:35.258170 30174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:35.258229 30172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:35.258172 30171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:35.258319 30166 server_base.cc:1034] running on GCE node
I20260618 23:30:35.258695 30166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:35.259023 30166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:35.260227 30166 hybrid_clock.cc:648] HybridClock initialized: now 1781825435260219 us; error 36 us; skew 500 ppm
Jun 18 23:30:35 dist-test-slave-47qt krb5kdc[30078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825435, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:35.262879 30166 init.cc:377] Logged in from keytab as kudu/127.8.39.189@KRBTEST.COM (short username kudu)
I20260618 23:30:35.263429 30166 webserver.cc:460] Webserver started at http://127.8.39.189:33719/ using document root <none> and password file <none>
I20260618 23:30:35.263693 30166 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:35.263770 30166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:35.263931 30166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:35.265551 30166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-1/data/instance:
uuid: "7673c595c8444ae7805bf2813deffd37"
format_stamp: "Formatted at 2026-06-18 23:30:35 on dist-test-slave-47qt"
server_key: "f75f300a49b44e88141d34428647d51c"
server_key_iv: "f80cffb3c4093a09fb1f3b01050cbde5"
server_key_version: "encryptionkey@0"
I20260618 23:30:35.265933 30166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-1/wal/instance:
uuid: "7673c595c8444ae7805bf2813deffd37"
format_stamp: "Formatted at 2026-06-18 23:30:35 on dist-test-slave-47qt"
server_key: "f75f300a49b44e88141d34428647d51c"
server_key_iv: "f80cffb3c4093a09fb1f3b01050cbde5"
server_key_version: "encryptionkey@0"
I20260618 23:30:35.267567 30166 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260618 23:30:35.268358 30181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:35.268568 30166 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:30:35.268678 30166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-1/wal
uuid: "7673c595c8444ae7805bf2813deffd37"
format_stamp: "Formatted at 2026-06-18 23:30:35 on dist-test-slave-47qt"
server_key: "f75f300a49b44e88141d34428647d51c"
server_key_iv: "f80cffb3c4093a09fb1f3b01050cbde5"
server_key_version: "encryptionkey@0"
I20260618 23:30:35.268757 30166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:35.273360 30159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44545 } attempt: 2
I20260618 23:30:35.273749 30161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.273633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:44545 (local address 127.0.0.1:54586)
0618 23:30:35.273689 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:35.273724 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.189:44545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260618 23:30:35.274065 30161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.273974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:44545 (local address 127.0.0.1:54600)
0618 23:30:35.274017 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:35.274040 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.189:44545: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260618 23:30:35.274144 30159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.39.189:44545: Network error: Client connection negotiation failed: client connection to 127.8.39.189:44545: connect: Connection refused (error 111)
I20260618 23:30:35.283425 30166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:35.286216 30166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:35.286485 30166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:35.290825 30166 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.189:44545
I20260618 23:30:35.290884 30233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.189:44545 every 8 connection(s)
I20260618 23:30:35.291278 30166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-1/data/info.pb
I20260618 23:30:35.291978 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 30166
I20260618 23:30:35.292129 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-1/wal/instance
I20260618 23:30:35.292436 8350 external_mini_cluster.cc:1428] Setting key dd751a20639e64a23e371e68ac6dff36
I20260618 23:30:35.292754 30234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:35.294895 30234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.39.190" port: 32995 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.39.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.188@KRBTEST.COM" created.
Jun 18 23:30:35 dist-test-slave-47qt krb5kdc[30078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825435, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.189@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:30:35.306887 30162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.295676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50240 (local address 127.8.39.190:32995)
0618 23:30:35.295752 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:35.295755 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:35.295936 (+ 181us) server_negotiation.cc:407] Connection header received
0618 23:30:35.296494 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:35.296496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:35.296512 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:35.296594 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:35.297318 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.297855 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.298946 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.299167 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.299259 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:35.303239 (+ 3980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:35.303240 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:35.303241 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:35.303250 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:35.305300 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:35.305722 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:35.305724 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:35.305724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:35.305795 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:35.306062 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:35.306064 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:35.306064 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:35.306228 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:35.306307 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:35.306658 (+ 351us) server_negotiation.cc:299] Negotiation successful
0618 23:30:35.306768 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260618 23:30:35.306913 30237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.295633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:32995 (local address 127.0.0.1:50240)
0618 23:30:35.295811 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:35.295849 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:35.296368 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:35.296678 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:35.296683 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:35.296945 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:35.297224 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:35.297228 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.297959 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.297960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:35.298831 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:35.298833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.299281 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.299282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:35.299425 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:35.299993 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:30:35.300002 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:35.303133 (+ 3131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:35.305401 (+ 2268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:35.305409 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:35.305411 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:35.305631 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:35.305874 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:35.305875 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:35.305875 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:35.305985 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:35.306356 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:35.306364 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:35.306512 (+ 148us) client_negotiation.cc:769] Sending connection context
0618 23:30:35.306610 (+ 98us) client_negotiation.cc:240] Negotiation successful
0618 23:30:35.306740 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
I20260618 23:30:35.307678 30234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44545 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 23:30:35 dist-test-slave-47qt krb5kdc[30078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825435, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.189@KRBTEST.COM for kudu/127.8.39.189@KRBTEST.COM
I20260618 23:30:35.316673 30237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.307970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:44545 (local address 127.0.0.1:54602)
0618 23:30:35.308020 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:35.308030 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:35.308070 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:35.308378 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:35.308379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:35.308517 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:35.308562 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:35.308564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.309460 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.309461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:35.309957 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:35.309958 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.310447 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.310448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:35.310509 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:35.310942 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:30:35.310947 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:35.313098 (+ 2151us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:35.315644 (+ 2546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:35.315645 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:35.315645 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:35.315811 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:35.316080 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:35.316081 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:35.316081 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:35.316134 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:35.316387 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:35.316388 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:35.316425 (+ 37us) client_negotiation.cc:769] Sending connection context
0618 23:30:35.316531 (+ 106us) client_negotiation.cc:240] Negotiation successful
0618 23:30:35.316605 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260618 23:30:35.316892 30239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.308058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54602 (local address 127.8.39.189:44545)
0618 23:30:35.308226 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:35.308230 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:35.308243 (+ 13us) server_negotiation.cc:407] Connection header received
0618 23:30:35.308259 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:35.308261 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:35.308293 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:35.308378 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:35.308639 (+ 261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.309363 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.310053 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.310359 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.310450 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:35.313339 (+ 2889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:35.313346 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:35.313346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:35.313352 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:35.315550 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:35.315943 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:35.315946 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:35.315946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:35.316014 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:35.316187 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:35.316188 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:35.316188 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:35.316319 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:35.316412 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:35.316670 (+ 258us) server_negotiation.cc:299] Negotiation successful
0618 23:30:35.316738 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":101,"threads_started":1}
I20260618 23:30:35.318202 30234 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.39.188" port: 36161 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:30:35.318624 30237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.318457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:36161 (local address 127.0.0.1:59542)
0618 23:30:35.318516 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:35.318590 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:36161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260618 23:30:35.318789 30182 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.39.188:36161: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.188.keytab.
Entry for principal kudu/127.8.39.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.188.keytab.
I20260618 23:30:35.319267 30237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.319175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:36161 (local address 127.0.0.1:59556)
0618 23:30:35.319213 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:35.319235 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:36161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260618 23:30:35.319423 30234 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.39.188:36161: Network error: Client connection negotiation failed: client connection to 127.8.39.188:36161: connect: Connection refused (error 111)
I20260618 23:30:35.319861 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.188:36161
--webserver_interface=127.8.39.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.188.keytab
--principal=kudu/127.8.39.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:35223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:32995,127.8.39.189:44545,127.8.39.188:36161
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:30:35.365337 30159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44545 } attempt: 3
I20260618 23:30:35.368646 30234 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.39.188" port: 36161 } attempt: 1
I20260618 23:30:35.369043 30237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.368924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:36161 (local address 127.0.0.1:59566)
0618 23:30:35.368968 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:35.369008 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:36161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260618 23:30:35.369398 30237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.369286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:36161 (local address 127.0.0.1:59572)
0618 23:30:35.369342 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:35.369371 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:36161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260618 23:30:35.369495 30234 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.39.188:36161: Network error: Client connection negotiation failed: client connection to 127.8.39.188:36161: connect: Connection refused (error 111)
Jun 18 23:30:35 dist-test-slave-47qt krb5kdc[30078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825435, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for kudu/127.8.39.189@KRBTEST.COM
I20260618 23:30:35.373620 30239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.365847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54610 (local address 127.8.39.189:44545)
0618 23:30:35.365919 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:35.365921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:35.365930 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:30:35.365955 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:35.365956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:35.365965 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:35.366027 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:35.366478 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.366956 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.368076 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.368327 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.368384 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:35.370808 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:35.370809 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:35.370810 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:35.370816 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:35.372427 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:35.372778 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:35.372779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:35.372779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:35.372837 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:35.373037 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:35.373038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:35.373038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:35.373228 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:35.373308 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:35.373460 (+ 152us) server_negotiation.cc:299] Negotiation successful
0618 23:30:35.373533 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260618 23:30:35.373625 30161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.365771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:44545 (local address 127.0.0.1:54610)
0618 23:30:35.365837 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:35.365850 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:35.365893 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:35.366056 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:35.366057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:35.366256 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:35.366388 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:35.366390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.367045 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.367045 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:35.367981 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:35.367984 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.368405 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.368406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:35.368484 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:35.369068 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:30:35.369073 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:35.370699 (+ 1626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:35.372504 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:35.372505 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:35.372506 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:35.372688 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:35.372904 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:35.372905 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:35.372906 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:35.372970 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:35.373298 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:35.373299 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:35.373347 (+ 48us) client_negotiation.cc:769] Sending connection context
0618 23:30:35.373435 (+ 88us) client_negotiation.cc:240] Negotiation successful
0618 23:30:35.373533 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260618 23:30:35.374053 30159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.39.188" port: 36161 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:30:35.374380 30161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.374286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:36161 (local address 127.0.0.1:59576)
0618 23:30:35.374334 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:35.374359 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:36161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260618 23:30:35.374701 30161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.374635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:36161 (local address 127.0.0.1:59588)
0618 23:30:35.374665 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:35.374689 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:36161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260618 23:30:35.374794 30159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.39.188:36161: Network error: Client connection negotiation failed: client connection to 127.8.39.188:36161: connect: Connection refused (error 111)
W20260618 23:30:35.402451 30241 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260618 23:30:35.402658 30241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:35.402676 30241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:35.402696 30241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:35.403728 30241 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:30:35.403777 30241 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:35.403791 30241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:35.403805 30241 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:30:35.403816 30241 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:30:35.405211 30241 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:35223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:32995,127.8.39.189:44545,127.8.39.188:36161
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.188.keytab
--principal=kudu/127.8.39.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.188:36161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.39.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30241
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:35.405398 30241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:35.405558 30241 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260618 23:30:35.405946 30159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.39.188" port: 36161 } attempt: 1
I20260618 23:30:35.406466 30161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.406314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:36161 (local address 127.0.0.1:59600)
0618 23:30:35.406398 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:35.406439 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:36161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260618 23:30:35.406905 30161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.406799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:36161 (local address 127.0.0.1:59602)
0618 23:30:35.406828 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:35.406860 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:36161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260618 23:30:35.407013 30159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.39.188:36161: Network error: Client connection negotiation failed: client connection to 127.8.39.188:36161: connect: Connection refused (error 111)
W20260618 23:30:35.410537 30246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:35.410526 30249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:35.410511 30247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:35.410720 30241 server_base.cc:1034] running on GCE node
I20260618 23:30:35.410969 30241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:35.411249 30241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:35.412421 30241 hybrid_clock.cc:648] HybridClock initialized: now 1781825435412399 us; error 51 us; skew 500 ppm
Jun 18 23:30:35 dist-test-slave-47qt krb5kdc[30078](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825435, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:35.414924 30241 init.cc:377] Logged in from keytab as kudu/127.8.39.188@KRBTEST.COM (short username kudu)
I20260618 23:30:35.415361 30241 webserver.cc:460] Webserver started at http://127.8.39.188:36191/ using document root <none> and password file <none>
I20260618 23:30:35.415551 30241 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:35.415592 30241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:35.415678 30241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:35.416637 30241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-2/data/instance:
uuid: "e916f05f8ac54187bbac6788e394fcd0"
format_stamp: "Formatted at 2026-06-18 23:30:35 on dist-test-slave-47qt"
server_key: "3d8d7ee225489a3b3c4be5ad176d0573"
server_key_iv: "c14afb3cff6f078813ed8ff1a83a54e8"
server_key_version: "encryptionkey@0"
I20260618 23:30:35.416942 30241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-2/wal/instance:
uuid: "e916f05f8ac54187bbac6788e394fcd0"
format_stamp: "Formatted at 2026-06-18 23:30:35 on dist-test-slave-47qt"
server_key: "3d8d7ee225489a3b3c4be5ad176d0573"
server_key_iv: "c14afb3cff6f078813ed8ff1a83a54e8"
server_key_version: "encryptionkey@0"
I20260618 23:30:35.418273 30241 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260618 23:30:35.419097 30256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:35.419405 30241 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:30:35.419521 30241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-2/wal
uuid: "e916f05f8ac54187bbac6788e394fcd0"
format_stamp: "Formatted at 2026-06-18 23:30:35 on dist-test-slave-47qt"
server_key: "3d8d7ee225489a3b3c4be5ad176d0573"
server_key_iv: "c14afb3cff6f078813ed8ff1a83a54e8"
server_key_version: "encryptionkey@0"
I20260618 23:30:35.419598 30241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:35.431751 30241 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:35.434115 30241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:35.434321 30241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:35.437656 30234 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.39.188" port: 36161 } attempt: 2
I20260618 23:30:35.438148 30237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.438009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:36161 (local address 127.0.0.1:59604)
0618 23:30:35.438070 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:35.438117 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:36161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260618 23:30:35.438560 30237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.438449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:36161 (local address 127.0.0.1:59612)
0618 23:30:35.438501 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:35.438530 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:36161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260618 23:30:35.438676 30234 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.39.188:36161: Network error: Client connection negotiation failed: client connection to 127.8.39.188:36161: connect: Connection refused (error 111)
I20260618 23:30:35.438679 30241 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.188:36161
I20260618 23:30:35.438719 30308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.188:36161 every 8 connection(s)
I20260618 23:30:35.439061 30241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-2/data/info.pb
I20260618 23:30:35.439992 30309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:35.441596 30309 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.39.190" port: 32995 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:30:35.445436 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 30241
I20260618 23:30:35.445552 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781825053542692-8350-0/minicluster-data/master-2/wal/instance
I20260618 23:30:35.445753 8350 external_mini_cluster.cc:1428] Setting key 17a754c80f62b0111661cf873d472f59
Jun 18 23:30:35 dist-test-slave-47qt krb5kdc[30078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825435, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.188@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
Jun 18 23:30:35 dist-test-slave-47qt krb5kdc[30078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.188@KRBTEST.COM
I20260618 23:30:35.453047 30162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.442359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50246 (local address 127.8.39.190:32995)
0618 23:30:35.442450 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:35.442452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:35.442534 (+ 82us) server_negotiation.cc:407] Connection header received
0618 23:30:35.443013 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:35.443014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:35.443023 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:35.443076 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:35.443668 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.444127 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.445295 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.445605 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.445691 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:35.448676 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:35.448678 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:35.448678 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:35.448685 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:35.450737 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:35.451531 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:35.451533 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:35.451533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:35.451603 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:35.452076 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:35.452077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:35.452077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:35.452228 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:35.452315 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:35.452783 (+ 468us) server_negotiation.cc:299] Negotiation successful
0618 23:30:35.452912 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 23:30:35.453122 30311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.442246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:32995 (local address 127.0.0.1:50246)
0618 23:30:35.442414 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:35.442460 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:35.442931 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:35.443140 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:35.443143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:35.443398 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:35.443595 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:35.443598 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.444204 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.444205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:35.445073 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:35.445076 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.445779 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.445781 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:35.445972 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:35.446577 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:30:35.446588 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:35.448545 (+ 1957us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:35.450898 (+ 2353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:35.450917 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:35.450920 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:35.451373 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:35.451730 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:35.451733 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:35.451734 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:35.451964 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:35.452345 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:35.452354 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:35.452586 (+ 232us) client_negotiation.cc:769] Sending connection context
0618 23:30:35.452694 (+ 108us) client_negotiation.cc:240] Negotiation successful
0618 23:30:35.452914 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
I20260618 23:30:35.454025 30309 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44545 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:30:35.457266 30313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.446370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59620 (local address 127.8.39.188:36161)
0618 23:30:35.446580 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:35.446584 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:35.446599 (+ 15us) server_negotiation.cc:407] Connection header received
0618 23:30:35.446658 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:35.446660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:35.446675 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:35.446738 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:35.447500 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.448464 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.449283 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.449726 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.449818 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:35.453585 (+ 3767us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:35.453613 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:35.453614 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:35.453623 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:35.455846 (+ 2223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:35.456282 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:35.456284 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:35.456285 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:35.456344 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:35.456666 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:35.456667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:35.456667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:35.456819 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:35.456992 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:35.457045 (+ 53us) server_negotiation.cc:299] Negotiation successful
0618 23:30:35.457156 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":110,"threads_started":1}
Jun 18 23:30:35 dist-test-slave-47qt krb5kdc[30078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825435, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.188@KRBTEST.COM for kudu/127.8.39.189@KRBTEST.COM
I20260618 23:30:35.462056 30311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.454367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:44545 (local address 127.0.0.1:54618)
0618 23:30:35.454426 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:35.454440 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:35.454481 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:35.454670 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:35.454671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:35.454834 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:35.454890 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:35.454892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.455942 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.455943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:35.456815 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:35.456817 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.457351 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.457353 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:35.457430 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:35.457807 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:30:35.457812 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:35.459280 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:35.460998 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:35.460999 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:35.460999 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:35.461276 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:35.461497 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:35.461498 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:35.461499 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:35.461568 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:35.461841 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:35.461842 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:35.461875 (+ 33us) client_negotiation.cc:769] Sending connection context
0618 23:30:35.461925 (+ 50us) client_negotiation.cc:240] Negotiation successful
0618 23:30:35.461993 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260618 23:30:35.462162 30239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.454459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54618 (local address 127.8.39.189:44545)
0618 23:30:35.454522 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:35.454524 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:35.454533 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:30:35.454559 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:35.454560 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:35.454571 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:35.454655 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:35.454991 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.455846 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.456897 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.457264 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.457356 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:35.459361 (+ 2005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:35.459362 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:35.459362 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:35.459370 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:35.460909 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:35.461366 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:35.461367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:35.461367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:35.461414 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:35.461640 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:35.461641 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:35.461642 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:35.461772 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:35.461846 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:35.461953 (+ 107us) server_negotiation.cc:299] Negotiation successful
0618 23:30:35.462041 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260618 23:30:35.462476 30309 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.39.188" port: 36161 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 23:30:35 dist-test-slave-47qt krb5kdc[30078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
Jun 18 23:30:35 dist-test-slave-47qt krb5kdc[30078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825435, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.188@KRBTEST.COM for kudu/127.8.39.188@KRBTEST.COM
I20260618 23:30:35.466537 30162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.458288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50250 (local address 127.8.39.190:32995)
0618 23:30:35.458365 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:35.458367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:35.458400 (+ 33us) server_negotiation.cc:407] Connection header received
0618 23:30:35.458481 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:35.458482 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:35.458522 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:35.458587 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:35.459118 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.459804 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.460646 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.460991 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.461053 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:35.463722 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:35.463723 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:35.463724 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:35.463731 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:35.465402 (+ 1671us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:35.465761 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:35.465762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:35.465763 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:35.465821 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:35.466017 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:35.466018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:35.466019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:35.466133 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:35.466308 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:35.466344 (+ 36us) server_negotiation.cc:299] Negotiation successful
0618 23:30:35.466418 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260618 23:30:35.469830 30311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.462740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:36161 (local address 127.0.0.1:59622)
0618 23:30:35.462772 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:35.462783 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:35.462817 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:35.462955 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:35.462955 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:35.463062 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:35.463090 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:35.463091 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.463825 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.463826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:35.464452 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:35.464454 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.465065 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.465066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:35.465109 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:35.465553 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:30:35.465558 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:35.466664 (+ 1106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:35.468749 (+ 2085us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:35.468750 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:35.468750 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:35.468974 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:35.469216 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:35.469217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:35.469218 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:35.469282 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:35.469496 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:35.469497 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:35.469538 (+ 41us) client_negotiation.cc:769] Sending connection context
0618 23:30:35.469611 (+ 73us) client_negotiation.cc:240] Negotiation successful
0618 23:30:35.469702 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260618 23:30:35.469908 30313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.462810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59622 (local address 127.8.39.188:36161)
0618 23:30:35.462845 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:35.462847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:35.462855 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:30:35.462874 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:35.462875 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:35.462884 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:35.462947 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:35.463161 (+ 214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.463728 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.464601 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.464977 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.465056 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:35.466869 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:35.466870 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:35.466871 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:35.466878 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:35.468661 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:35.469050 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:35.469050 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:35.469051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:35.469100 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:35.469344 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:35.469344 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:35.469345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:35.469437 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:35.469483 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:35.469627 (+ 144us) server_negotiation.cc:299] Negotiation successful
0618 23:30:35.469742 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":19}
I20260618 23:30:35.471716 30309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e916f05f8ac54187bbac6788e394fcd0: Bootstrap starting.
Jun 18 23:30:35 dist-test-slave-47qt krb5kdc[30078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.189@KRBTEST.COM
I20260618 23:30:35.472795 30309 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e916f05f8ac54187bbac6788e394fcd0: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:35.473165 30309 log.cc:824] T 00000000000000000000000000000000 P e916f05f8ac54187bbac6788e394fcd0: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:35.474318 30309 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e916f05f8ac54187bbac6788e394fcd0: No bootstrap required, opened a new log
I20260618 23:30:35.476287 30239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.467193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54620 (local address 127.8.39.189:44545)
0618 23:30:35.467231 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:35.467232 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:35.467239 (+ 7us) server_negotiation.cc:407] Connection header received
0618 23:30:35.467262 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:35.467263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:35.467272 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:35.467337 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:35.467922 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.468834 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.469823 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.470123 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.470198 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:35.473022 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:35.473023 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:35.473024 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:35.473031 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:35.474844 (+ 1813us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:35.475437 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:35.475438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:35.475438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:35.475501 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:35.475799 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:35.475800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:35.475800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:35.475918 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:35.476035 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:35.476164 (+ 129us) server_negotiation.cc:299] Negotiation successful
0618 23:30:35.476226 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260618 23:30:35.477468 30309 raft_consensus.cc:348] T 00000000000000000000000000000000 P e916f05f8ac54187bbac6788e394fcd0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fab5f31acc1a4190b59546b902a58998" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 32995 } } peers { permanent_uuid: "7673c595c8444ae7805bf2813deffd37" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44545 } } peers { permanent_uuid: "e916f05f8ac54187bbac6788e394fcd0" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 36161 } }
I20260618 23:30:35.477661 30309 raft_consensus.cc:729] T 00000000000000000000000000000000 P e916f05f8ac54187bbac6788e394fcd0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e916f05f8ac54187bbac6788e394fcd0, State: Initialized, Role: FOLLOWER
I20260618 23:30:35.477761 30309 consensus_queue.cc:260] T 00000000000000000000000000000000 P e916f05f8ac54187bbac6788e394fcd0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fab5f31acc1a4190b59546b902a58998" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 32995 } } peers { permanent_uuid: "7673c595c8444ae7805bf2813deffd37" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44545 } } peers { permanent_uuid: "e916f05f8ac54187bbac6788e394fcd0" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 36161 } }
I20260618 23:30:35.477888 8350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:30:35.478121 30309 sys_catalog.cc:564] T 00000000000000000000000000000000 P e916f05f8ac54187bbac6788e394fcd0 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:30:35.478087 30315 sys_catalog.cc:455] T 00000000000000000000000000000000 P e916f05f8ac54187bbac6788e394fcd0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fab5f31acc1a4190b59546b902a58998" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 32995 } } peers { permanent_uuid: "7673c595c8444ae7805bf2813deffd37" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44545 } } peers { permanent_uuid: "e916f05f8ac54187bbac6788e394fcd0" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 36161 } } }
I20260618 23:30:35.478197 30315 sys_catalog.cc:458] T 00000000000000000000000000000000 P e916f05f8ac54187bbac6788e394fcd0 [sys.catalog]: This master's current role is: FOLLOWER
W20260618 23:30:35.479995 30326 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e916f05f8ac54187bbac6788e394fcd0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 23:30:35.480047 30326 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:30:35.479092 30133 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "fab5f31acc1a4190b59546b902a58998"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50250
I20260618 23:30:35.482141 30159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.39.188" port: 36161 } attempt: 2
I20260618 23:30:35.480262 30241 master_runner.cc:186] Error getting master registration for 127.8.39.190:32995: OK, instance_id { permanent_uuid: "fab5f31acc1a4190b59546b902a58998" instance_seqno: 1781825435136745 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260618 23:30:35.482460 30241 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Jun 18 23:30:35 dist-test-slave-47qt krb5kdc[30078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825435, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for kudu/127.8.39.188@KRBTEST.COM
I20260618 23:30:35.490062 30161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.482367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:36161 (local address 127.0.0.1:59638)
0618 23:30:35.482412 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:35.482423 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:35.482460 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:35.482734 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:35.482735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:35.482908 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:35.482953 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:35.482955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.483910 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.483912 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:35.484797 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:35.484800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.485347 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.485348 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:35.485402 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:35.485858 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:30:35.485864 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:35.487235 (+ 1371us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:35.488896 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:35.488897 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:35.488898 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:35.489144 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:35.489352 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:35.489353 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:35.489353 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:35.489419 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:35.489737 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:35.489738 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:35.489788 (+ 50us) client_negotiation.cc:769] Sending connection context
0618 23:30:35.489865 (+ 77us) client_negotiation.cc:240] Negotiation successful
0618 23:30:35.489959 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260618 23:30:35.490063 30313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.482541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59638 (local address 127.8.39.188:36161)
0618 23:30:35.482584 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:35.482586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:35.482593 (+ 7us) server_negotiation.cc:407] Connection header received
0618 23:30:35.482613 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:35.482613 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:35.482622 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:35.482668 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:35.483051 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.483811 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.484907 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.485262 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.485347 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:35.487381 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:35.487382 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:35.487383 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:35.487389 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:35.488795 (+ 1406us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:35.489219 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:35.489220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:35.489221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:35.489277 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:35.489494 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:35.489495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:35.489496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:35.489663 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:35.489729 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:35.489899 (+ 170us) server_negotiation.cc:299] Negotiation successful
0618 23:30:35.489972 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260618 23:30:35.491638 30159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fab5f31acc1a4190b59546b902a58998: Bootstrap starting.
I20260618 23:30:35.492292 30159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fab5f31acc1a4190b59546b902a58998: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:35.492581 30159 log.cc:824] T 00000000000000000000000000000000 P fab5f31acc1a4190b59546b902a58998: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:35.493398 30159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fab5f31acc1a4190b59546b902a58998: No bootstrap required, opened a new log
I20260618 23:30:35.494427 30159 raft_consensus.cc:348] T 00000000000000000000000000000000 P fab5f31acc1a4190b59546b902a58998 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fab5f31acc1a4190b59546b902a58998" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 32995 } } peers { permanent_uuid: "7673c595c8444ae7805bf2813deffd37" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44545 } } peers { permanent_uuid: "e916f05f8ac54187bbac6788e394fcd0" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 36161 } }
I20260618 23:30:35.494558 30159 raft_consensus.cc:729] T 00000000000000000000000000000000 P fab5f31acc1a4190b59546b902a58998 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fab5f31acc1a4190b59546b902a58998, State: Initialized, Role: FOLLOWER
I20260618 23:30:35.494652 30159 consensus_queue.cc:260] T 00000000000000000000000000000000 P fab5f31acc1a4190b59546b902a58998 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fab5f31acc1a4190b59546b902a58998" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 32995 } } peers { permanent_uuid: "7673c595c8444ae7805bf2813deffd37" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44545 } } peers { permanent_uuid: "e916f05f8ac54187bbac6788e394fcd0" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 36161 } }
I20260618 23:30:35.494948 30329 sys_catalog.cc:455] T 00000000000000000000000000000000 P fab5f31acc1a4190b59546b902a58998 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fab5f31acc1a4190b59546b902a58998" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 32995 } } peers { permanent_uuid: "7673c595c8444ae7805bf2813deffd37" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44545 } } peers { permanent_uuid: "e916f05f8ac54187bbac6788e394fcd0" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 36161 } } }
I20260618 23:30:35.495000 30159 sys_catalog.cc:564] T 00000000000000000000000000000000 P fab5f31acc1a4190b59546b902a58998 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:30:35.495040 30133 raft_consensus.cc:482] T 00000000000000000000000000000000 P fab5f31acc1a4190b59546b902a58998 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260618 23:30:35.495345 30133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fab5f31acc1a4190b59546b902a58998 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:35.496160 30133 raft_consensus.cc:504] T 00000000000000000000000000000000 P fab5f31acc1a4190b59546b902a58998 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fab5f31acc1a4190b59546b902a58998" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 32995 } } peers { permanent_uuid: "7673c595c8444ae7805bf2813deffd37" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44545 } } peers { permanent_uuid: "e916f05f8ac54187bbac6788e394fcd0" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 36161 } }
I20260618 23:30:35.497110 30133 leader_election.cc:288] T 00000000000000000000000000000000 P fab5f31acc1a4190b59546b902a58998 [CANDIDATE]: Term 1 election: Requested vote from peers 7673c595c8444ae7805bf2813deffd37 (127.8.39.189:44545), e916f05f8ac54187bbac6788e394fcd0 (127.8.39.188:36161)
I20260618 23:30:35.497438 30283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fab5f31acc1a4190b59546b902a58998" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "e916f05f8ac54187bbac6788e394fcd0"
I20260618 23:30:35.497568 30283 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e916f05f8ac54187bbac6788e394fcd0 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:35.498690 30283 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e916f05f8ac54187bbac6788e394fcd0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fab5f31acc1a4190b59546b902a58998 in term 1.
I20260618 23:30:35.498919 30329 sys_catalog.cc:458] T 00000000000000000000000000000000 P fab5f31acc1a4190b59546b902a58998 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:30:35.498943 30107 leader_election.cc:302] T 00000000000000000000000000000000 P fab5f31acc1a4190b59546b902a58998 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e916f05f8ac54187bbac6788e394fcd0, fab5f31acc1a4190b59546b902a58998; no voters:
I20260618 23:30:35.499495 30329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fab5f31acc1a4190b59546b902a58998 [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:30:35.499652 30329 raft_consensus.cc:686] T 00000000000000000000000000000000 P fab5f31acc1a4190b59546b902a58998 [term 1 LEADER]: Becoming Leader. State: Replica: fab5f31acc1a4190b59546b902a58998, State: Running, Role: LEADER
I20260618 23:30:35.499734 30329 consensus_queue.cc:237] T 00000000000000000000000000000000 P fab5f31acc1a4190b59546b902a58998 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fab5f31acc1a4190b59546b902a58998" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 32995 } } peers { permanent_uuid: "7673c595c8444ae7805bf2813deffd37" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44545 } } peers { permanent_uuid: "e916f05f8ac54187bbac6788e394fcd0" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 36161 } }
I20260618 23:30:35.501029 30348 sys_catalog.cc:455] T 00000000000000000000000000000000 P fab5f31acc1a4190b59546b902a58998 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fab5f31acc1a4190b59546b902a58998. Latest consensus state: current_term: 1 leader_uuid: "fab5f31acc1a4190b59546b902a58998" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fab5f31acc1a4190b59546b902a58998" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 32995 } } peers { permanent_uuid: "7673c595c8444ae7805bf2813deffd37" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44545 } } peers { permanent_uuid: "e916f05f8ac54187bbac6788e394fcd0" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 36161 } } }
I20260618 23:30:35.501152 30348 sys_catalog.cc:458] T 00000000000000000000000000000000 P fab5f31acc1a4190b59546b902a58998 [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:35.500993 30091 master_runner.cc:186] Error getting master registration for 127.8.39.189:44545: OK, instance_id { permanent_uuid: "7673c595c8444ae7805bf2813deffd37" instance_seqno: 1781825435289586 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260618 23:30:35.503741 30091 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260618 23:30:35.508087 30162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.498265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50258 (local address 127.8.39.190:32995)
0618 23:30:35.498307 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:35.498309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:35.499064 (+ 755us) server_negotiation.cc:407] Connection header received
0618 23:30:35.499146 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:35.499147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:35.499157 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:35.499202 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:35.500135 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.500833 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.501710 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.501987 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.502056 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:35.503994 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:35.503995 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:35.503996 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:35.504004 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:35.506711 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:35.507094 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:35.507095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:35.507095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:35.507201 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:35.507480 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:35.507481 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:35.507481 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:35.507608 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:35.507718 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:35.507824 (+ 106us) server_negotiation.cc:299] Negotiation successful
0618 23:30:35.507902 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260618 23:30:35.509390 30347 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fab5f31acc1a4190b59546b902a58998: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260618 23:30:35.509773 30313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.498408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59642 (local address 127.8.39.188:36161)
0618 23:30:35.498443 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:35.498445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:35.498517 (+ 72us) server_negotiation.cc:407] Connection header received
0618 23:30:35.498619 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:35.498620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:35.498631 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:35.498677 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:35.500035 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.500782 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.502531 (+ 1749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.502848 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.502894 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:35.505733 (+ 2839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:35.505735 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:35.505735 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:35.505743 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:35.507539 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:35.507934 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:35.507935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:35.507936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:35.507995 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:35.508410 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:35.508411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:35.508411 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:35.508537 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:35.508600 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:35.509575 (+ 975us) server_negotiation.cc:299] Negotiation successful
0618 23:30:35.509655 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
W20260618 23:30:35.510174 30347 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:30:35.513301 30239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.498692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54626 (local address 127.8.39.189:44545)
0618 23:30:35.498741 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:35.498743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:35.503285 (+ 4542us) server_negotiation.cc:407] Connection header received
0618 23:30:35.503400 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:35.503401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:35.503413 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:35.503448 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:35.505679 (+ 2231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.506421 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.508444 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.508753 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.508794 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:35.510323 (+ 1529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:35.510324 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:35.510325 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:35.510332 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:35.512042 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:35.512434 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:35.512435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:35.512435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:35.512504 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:35.512708 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:35.512709 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:35.512709 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:35.512841 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:35.512928 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:35.513061 (+ 133us) server_negotiation.cc:299] Negotiation successful
0618 23:30:35.513197 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260618 23:30:35.521240 30350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:30:35.521646 30350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:30:35.523859 30283 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e916f05f8ac54187bbac6788e394fcd0 [term 1 FOLLOWER]: Refusing update from remote peer fab5f31acc1a4190b59546b902a58998: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:30:35.524166 30329 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fab5f31acc1a4190b59546b902a58998 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e916f05f8ac54187bbac6788e394fcd0" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 36161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:30:35.525439 30315 sys_catalog.cc:455] T 00000000000000000000000000000000 P e916f05f8ac54187bbac6788e394fcd0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fab5f31acc1a4190b59546b902a58998. Latest consensus state: current_term: 1 leader_uuid: "fab5f31acc1a4190b59546b902a58998" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fab5f31acc1a4190b59546b902a58998" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 32995 } } peers { permanent_uuid: "7673c595c8444ae7805bf2813deffd37" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44545 } } peers { permanent_uuid: "e916f05f8ac54187bbac6788e394fcd0" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 36161 } } }
I20260618 23:30:35.525532 30315 sys_catalog.cc:458] T 00000000000000000000000000000000 P e916f05f8ac54187bbac6788e394fcd0 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:30:35.526536 30348 sys_catalog.cc:455] T 00000000000000000000000000000000 P fab5f31acc1a4190b59546b902a58998 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fab5f31acc1a4190b59546b902a58998" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fab5f31acc1a4190b59546b902a58998" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 32995 } } peers { permanent_uuid: "7673c595c8444ae7805bf2813deffd37" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44545 } } peers { permanent_uuid: "e916f05f8ac54187bbac6788e394fcd0" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 36161 } } }
I20260618 23:30:35.526628 30348 sys_catalog.cc:458] T 00000000000000000000000000000000 P fab5f31acc1a4190b59546b902a58998 [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:35.526708 30352 mvcc.cc:205] Tried to move back new op lower bound from 7298356983902736384 to 7298356983807868928. Current Snapshot: MvccSnapshot[applied={T|T < 7298356983902736384}]
I20260618 23:30:35.527135 30315 sys_catalog.cc:455] T 00000000000000000000000000000000 P e916f05f8ac54187bbac6788e394fcd0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fab5f31acc1a4190b59546b902a58998" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fab5f31acc1a4190b59546b902a58998" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 32995 } } peers { permanent_uuid: "7673c595c8444ae7805bf2813deffd37" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44545 } } peers { permanent_uuid: "e916f05f8ac54187bbac6788e394fcd0" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 36161 } } }
I20260618 23:30:35.527225 30315 sys_catalog.cc:458] T 00000000000000000000000000000000 P e916f05f8ac54187bbac6788e394fcd0 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:30:35.527253 30350 catalog_manager.cc:1347] Generated new cluster ID: 50963f8591134461b2c6a74756b80825
I20260618 23:30:35.527307 30350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:30:35.527463 30353 mvcc.cc:205] Tried to move back new op lower bound from 7298356983902736384 to 7298356983807868928. Current Snapshot: MvccSnapshot[applied={T|T < 7298356983902736384}]
I20260618 23:30:35.529803 30234 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.39.188" port: 36161 } attempt: 3
Jun 18 23:30:35 dist-test-slave-47qt krb5kdc[30078](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825435, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.189@KRBTEST.COM for kudu/127.8.39.188@KRBTEST.COM
I20260618 23:30:35.538810 30313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.530214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59648 (local address 127.8.39.188:36161)
0618 23:30:35.530247 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:35.530248 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:35.530254 (+ 6us) server_negotiation.cc:407] Connection header received
0618 23:30:35.530272 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:35.530273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:35.530281 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:35.530321 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:35.530663 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.531389 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.532544 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.533021 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.533108 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:35.535834 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:35.535835 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:35.535835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:35.535842 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:35.537546 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:35.537924 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:35.537925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:35.537926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:35.537984 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:35.538185 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:35.538185 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:35.538185 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:35.538315 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:35.538364 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:35.538621 (+ 257us) server_negotiation.cc:299] Negotiation successful
0618 23:30:35.538717 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260618 23:30:35.539372 30237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:35.530040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:36161 (local address 127.0.0.1:59648)
0618 23:30:35.530112 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:35.530122 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:35.530154 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:35.530373 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:35.530374 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:35.530549 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:35.530591 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:35.530592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.531480 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.531481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:35.532424 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:35.532427 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:35.533218 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:35.533220 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:35.533546 (+ 326us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:35.534273 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:30:35.534279 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:35.535738 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:35.537640 (+ 1902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:35.537641 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:35.537642 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:35.537846 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:35.538051 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:35.538052 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:35.538052 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:35.538112 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:35.538399 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:35.538400 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:35.538465 (+ 65us) client_negotiation.cc:769] Sending connection context
0618 23:30:35.538557 (+ 92us) client_negotiation.cc:240] Negotiation successful
0618 23:30:35.539214 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260618 23:30:35.541081 30234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7673c595c8444ae7805bf2813deffd37: Bootstrap starting.
I20260618 23:30:35.541770 30234 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7673c595c8444ae7805bf2813deffd37: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:35.542057 30234 log.cc:824] T 00000000000000000000000000000000 P 7673c595c8444ae7805bf2813deffd37: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:35.548775 30234 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7673c595c8444ae7805bf2813deffd37: No bootstrap required, opened a new log
I20260618 23:30:35.552980 30234 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7673c595c8444ae7805bf2813deffd37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fab5f31acc1a4190b59546b902a58998" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 32995 } } peers { permanent_uuid: "7673c595c8444ae7805bf2813deffd37" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44545 } } peers { permanent_uuid: "e916f05f8ac54187bbac6788e394fcd0" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 36161 } }
I20260618 23:30:35.553150 30234 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7673c595c8444ae7805bf2813deffd37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7673c595c8444ae7805bf2813deffd37, State: Initialized, Role: FOLLOWER
I20260618 23:30:35.553309 30234 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7673c595c8444ae7805bf2813deffd37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fab5f31acc1a4190b59546b902a58998" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 32995 } } peers { permanent_uuid: "7673c595c8444ae7805bf2813deffd37" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44545 } } peers { permanent_uuid: "e916f05f8ac54187bbac6788e394fcd0" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 36161 } }
I20260618 23:30:35.553653 30234 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7673c595c8444ae7805bf2813deffd37 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:30:35.553718 30205 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fab5f31acc1a4190b59546b902a58998" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7673c595c8444ae7805bf2813deffd37"
I20260618 23:30:35.553828 30205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7673c595c8444ae7805bf2813deffd37 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:35.554814 30205 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7673c595c8444ae7805bf2813deffd37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fab5f31acc1a4190b59546b902a58998 in term 1.
I20260618 23:30:35.555888 30208 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7673c595c8444ae7805bf2813deffd37 [term 1 FOLLOWER]: Refusing update from remote peer fab5f31acc1a4190b59546b902a58998: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:30:35.556150 30329 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fab5f31acc1a4190b59546b902a58998 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7673c595c8444ae7805bf2813deffd37" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260618 23:30:35.556844 30369 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7673c595c8444ae7805bf2813deffd37: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 23:30:35.556895 30369 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:30:35.559590 30358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7673c595c8444ae7805bf2813deffd37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fab5f31acc1a4190b59546b902a58998" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fab5f31acc1a4190b59546b902a58998" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 32995 } } peers { permanent_uuid: "7673c595c8444ae7805bf2813deffd37" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44545 } } peers { permanent_uuid: "e916f05f8ac54187bbac6788e394fcd0" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 36161 } } }
I20260618 23:30:35.559686 30358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7673c595c8444ae7805bf2813deffd37 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:30:35.559741 30358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7673c595c8444ae7805bf2813deffd37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fab5f31acc1a4190b59546b902a58998. Latest consensus state: current_term: 1 leader_uuid: "fab5f31acc1a4190b59546b902a58998" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fab5f31acc1a4190b59546b902a58998" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 32995 } } peers { permanent_uuid: "7673c595c8444ae7805bf2813deffd37" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44545 } } peers { permanent_uuid: "e916f05f8ac54187bbac6788e394fcd0" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 36161 } } }
I20260618 23:30:35.559795 30358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7673c595c8444ae7805bf2813deffd37 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:30:35.559856 30358 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7673c595c8444ae7805bf2813deffd37 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fab5f31acc1a4190b59546b902a58998" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fab5f31acc1a4190b59546b902a58998" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 32995 } } peers { permanent_uuid: "7673c595c8444ae7805bf2813deffd37" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44545 } } peers { permanent_uuid: "e916f05f8ac54187bbac6788e394fcd0" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 36161 } } }
I20260618 23:30:35.559907 30358 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7673c595c8444ae7805bf2813deffd37 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:30:35.560045 30350 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:30:35.560752 30350 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:30:35.568346 30350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fab5f31acc1a4190b59546b902a58998: Generated new TSK 0
I20260618 23:30:35.568533 30350 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260618 23:30:35.571712 30329 sys_catalog.cc:455] T 00000000000000000000000000000000 P fab5f31acc1a4190b59546b902a58998 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fab5f31acc1a4190b59546b902a58998" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fab5f31acc1a4190b59546b902a58998" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 32995 } } peers { permanent_uuid: "7673c595c8444ae7805bf2813deffd37" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 44545 } } peers { permanent_uuid: "e916f05f8ac54187bbac6788e394fcd0" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 36161 } } }
I20260618 23:30:35.571816 30329 sys_catalog.cc:458] T 00000000000000000000000000000000 P fab5f31acc1a4190b59546b902a58998 [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:35.571944 30372 mvcc.cc:205] Tried to move back new op lower bound from 7298356983902736384 to 7298356983807868928. Current Snapshot: MvccSnapshot[applied={T|T < 7298356983902736384}]
W20260618 23:30:35.597470 30162 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:30:35.595801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50274 (local address 127.8.39.190:32995)
0618 23:30:35.595853 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:35.595856 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:35.596767 (+ 911us) server_negotiation.cc:407] Connection header received
0618 23:30:35.596871 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:35.596872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:35.596884 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:35.596915 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:35.597394 (+ 479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50274: BlockingRecv error: recv got EOF from 127.0.0.1:50274 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260618 23:30:35.598218 30239 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:30:35.596183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54628 (local address 127.8.39.189:44545)
0618 23:30:35.596239 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:35.596241 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:35.597737 (+ 1496us) server_negotiation.cc:407] Connection header received
0618 23:30:35.597823 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:35.597824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:35.597834 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:35.597863 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:35.598147 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54628: BlockingRecv error: recv got EOF from 127.0.0.1:54628 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260618 23:30:35.599077 30313 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:30:35.596514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59652 (local address 127.8.39.188:36161)
0618 23:30:35.596551 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:35.596553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:35.598443 (+ 1890us) server_negotiation.cc:407] Connection header received
0618 23:30:35.598537 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:35.598538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:35.598548 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:35.598575 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:35.598987 (+ 412us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59652: BlockingRecv error: recv got EOF from 127.0.0.1:59652 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260618 23:30:35.606376 30239 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:30:35.600935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54630 (local address 127.8.39.189:44545)
0618 23:30:35.601005 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:35.601008 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:35.605736 (+ 4728us) server_negotiation.cc:407] Connection header received
0618 23:30:35.605883 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:35.605884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:35.605906 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:35.605939 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:35.606321 (+ 382us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54630: BlockingRecv error: recv got EOF from 127.0.0.1:54630 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
W20260618 23:30:35.614454 30239 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:30:35.613534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54634 (local address 127.8.39.189:44545)
0618 23:30:35.613577 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:35.613579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:35.613868 (+ 289us) server_negotiation.cc:407] Connection header received
0618 23:30:35.613979 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:35.613980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:35.613992 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:35.614022 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:35.614395 (+ 373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54634: BlockingRecv error: recv got EOF from 127.0.0.1:54634 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260618 23:30:35.618551 30239 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:30:35.617666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54640 (local address 127.8.39.189:44545)
0618 23:30:35.617703 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:35.617705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:35.617963 (+ 258us) server_negotiation.cc:407] Connection header received
0618 23:30:35.618062 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:35.618063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:35.618073 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:35.618103 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:35.618497 (+ 394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54640: BlockingRecv error: recv got EOF from 127.0.0.1:54640 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260618 23:30:35.630391 30239 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:30:35.629503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54656 (local address 127.8.39.189:44545)
0618 23:30:35.629547 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:35.629549 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:35.629821 (+ 272us) server_negotiation.cc:407] Connection header received
0618 23:30:35.629928 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:35.629930 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:35.629941 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:35.629971 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:35.630332 (+ 361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54656: BlockingRecv error: recv got EOF from 127.0.0.1:54656 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260618 23:30:35.640516 30239 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:30:35.639656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54672 (local address 127.8.39.189:44545)
0618 23:30:35.639696 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:35.639698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:35.639956 (+ 258us) server_negotiation.cc:407] Connection header received
0618 23:30:35.640058 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:35.640059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:35.640071 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:35.640100 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:35.640459 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54672: BlockingRecv error: recv got EOF from 127.0.0.1:54672 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260618 23:30:35.658838 30239 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:30:35.657901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54688 (local address 127.8.39.189:44545)
0618 23:30:35.657945 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:35.657947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:35.658223 (+ 276us) server_negotiation.cc:407] Connection header received
0618 23:30:35.658336 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:35.658337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:35.658349 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:35.658381 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:35.658765 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54688: BlockingRecv error: recv got EOF from 127.0.0.1:54688 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260618 23:30:35.693455 30239 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:30:35.692458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54694 (local address 127.8.39.189:44545)
0618 23:30:35.692532 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:35.692536 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:35.692803 (+ 267us) server_negotiation.cc:407] Connection header received
0618 23:30:35.692896 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:35.692897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:35.692911 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:35.692976 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:35.693396 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54694: BlockingRecv error: recv got EOF from 127.0.0.1:54694 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
W20260618 23:30:35.759354 30239 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:30:35.758676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54706 (local address 127.8.39.189:44545)
0618 23:30:35.758773 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:35.758775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:35.758826 (+ 51us) server_negotiation.cc:407] Connection header received
0618 23:30:35.758888 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:35.758889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:35.758900 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:35.758948 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:35.759270 (+ 322us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54706: BlockingRecv error: recv got EOF from 127.0.0.1:54706 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20260618 23:30:35.889223 30239 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:30:35.888553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54712 (local address 127.8.39.189:44545)
0618 23:30:35.888650 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:35.888652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:35.888721 (+ 69us) server_negotiation.cc:407] Connection header received
0618 23:30:35.888750 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:35.888751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:35.888761 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:35.888826 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:35.889162 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54712: BlockingRecv error: recv got EOF from 127.0.0.1:54712 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
W20260618 23:30:36.147514 30239 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:30:36.146689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54724 (local address 127.8.39.189:44545)
0618 23:30:36.146795 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:36.146797 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:36.146836 (+ 39us) server_negotiation.cc:407] Connection header received
0618 23:30:36.146867 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:36.146868 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:36.146880 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:36.146935 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:36.147450 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54724: BlockingRecv error: recv got EOF from 127.0.0.1:54724 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260618 23:30:36.480861 30326 catalog_manager.cc:1259] Loaded cluster ID: 50963f8591134461b2c6a74756b80825
I20260618 23:30:36.481143 30326 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e916f05f8ac54187bbac6788e394fcd0: loading cluster ID for follower catalog manager: success
I20260618 23:30:36.482113 30326 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e916f05f8ac54187bbac6788e394fcd0: acquiring CA information for follower catalog manager: success
I20260618 23:30:36.482398 30326 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e916f05f8ac54187bbac6788e394fcd0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260618 23:30:36.557477 30369 catalog_manager.cc:1259] Loaded cluster ID: 50963f8591134461b2c6a74756b80825
I20260618 23:30:36.557549 30369 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7673c595c8444ae7805bf2813deffd37: loading cluster ID for follower catalog manager: success
I20260618 23:30:36.558398 30369 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7673c595c8444ae7805bf2813deffd37: acquiring CA information for follower catalog manager: success
I20260618 23:30:36.558712 30369 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7673c595c8444ae7805bf2813deffd37: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260618 23:30:36.663455 30441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:36.660691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54740 (local address 127.8.39.189:44545)
0618 23:30:36.660798 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:36.660800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:36.660809 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:30:36.660893 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:36.660893 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:36.660940 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:36.661012 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:30:36.661195 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:36.661677 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:36.662406 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:36.662594 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:36.662657 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:36.662781 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:30:36.662889 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:30:36.662950 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:36.663200 (+ 250us) server_negotiation.cc:299] Negotiation successful
0618 23:30:36.663252 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":60,"threads_started":1}
I20260618 23:30:36.668437 30447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:36.665541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59660 (local address 127.8.39.188:36161)
0618 23:30:36.665653 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:36.665655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:36.665664 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:30:36.665746 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:36.665747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:36.665780 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:36.665849 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:30:36.665993 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:36.666465 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:36.667559 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:36.667769 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:36.667831 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:36.668064 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:30:36.668160 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:30:36.668229 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:36.668342 (+ 113us) server_negotiation.cc:299] Negotiation successful
0618 23:30:36.668361 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":72,"threads_started":1}
I20260618 23:30:36.674182 30441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:36.670322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54746 (local address 127.8.39.189:44545)
0618 23:30:36.670372 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:36.670374 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:36.670532 (+ 158us) server_negotiation.cc:407] Connection header received
0618 23:30:36.670654 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:36.670656 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:36.670669 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:36.670728 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:30:36.670852 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:36.671667 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:36.673069 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:36.673407 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:36.673468 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:36.673762 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:30:36.673795 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:30:36.673853 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:36.674070 (+ 217us) server_negotiation.cc:299] Negotiation successful
0618 23:30:36.674095 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260618 23:30:36.674309 30447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:36.670523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59674 (local address 127.8.39.188:36161)
0618 23:30:36.670562 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:36.670563 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:36.670820 (+ 257us) server_negotiation.cc:407] Connection header received
0618 23:30:36.670901 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:36.670902 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:36.670910 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:36.670960 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:30:36.671077 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:36.671854 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:36.673334 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:36.673700 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:36.673775 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:36.674016 (+ 241us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:30:36.674049 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:30:36.674098 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:36.674211 (+ 113us) server_negotiation.cc:299] Negotiation successful
0618 23:30:36.674242 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260618 23:30:36.674844 30453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:36.670415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50286 (local address 127.8.39.190:32995)
0618 23:30:36.670621 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:36.670623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:36.670684 (+ 61us) server_negotiation.cc:407] Connection header received
0618 23:30:36.670762 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:36.670763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:36.670812 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:36.670903 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:30:36.671141 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:36.671887 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:36.673506 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:36.673792 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:36.673860 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:36.674053 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:30:36.674354 (+ 301us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:30:36.674532 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:36.674766 (+ 234us) server_negotiation.cc:299] Negotiation successful
0618 23:30:36.674787 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182,"thread_start_us":63,"threads_started":1}
I20260618 23:30:36.676716 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 30091
I20260618 23:30:36.682240 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 30166
I20260618 23:30:36.688905 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 30241
2026-06-18T23:30:36Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2888 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 18 23:30:36 dist-test-slave-47qt krb5kdc[30458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 18 23:30:36 dist-test-slave-47qt krb5kdc[30458](info): set up 2 sockets
Jun 18 23:30:36 dist-test-slave-47qt krb5kdc[30458](info): commencing operation
krb5kdc: starting...
W20260618 23:30:37.745421 8350 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/krb5kdc/test-user.keytab.
Jun 18 23:30:37 dist-test-slave-47qt krb5kdc[30458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-18T23:30:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-18T23:30:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.39.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
Entry for principal kudu/127.8.39.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab.
I20260618 23:30:37.864743 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:44193
--webserver_interface=127.8.39.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:42493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:44193,127.8.39.189:38543,127.8.39.188:42657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260618 23:30:37.947206 30471 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260618 23:30:37.947419 30471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:37.947436 30471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:37.947456 30471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:37.948500 30471 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:30:37.948535 30471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:37.948549 30471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:37.948562 30471 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:30:37.948575 30471 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:30:37.950064 30471 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:42493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:44193,127.8.39.189:38543,127.8.39.188:42657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.190.keytab
--principal=kudu/127.8.39.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.190:44193
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.39.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:37.950263 30471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:37.950438 30471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:37.955693 30476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:37.955673 30477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:37.955715 30471 server_base.cc:1034] running on GCE node
W20260618 23:30:37.955693 30479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:37.956166 30471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:37.956414 30471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:37.957547 30471 hybrid_clock.cc:648] HybridClock initialized: now 1781825437957527 us; error 37 us; skew 500 ppm
Jun 18 23:30:37 dist-test-slave-47qt krb5kdc[30458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825437, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:38.001639 30471 init.cc:377] Logged in from keytab as kudu/127.8.39.190@KRBTEST.COM (short username kudu)
I20260618 23:30:38.002195 30471 webserver.cc:460] Webserver started at http://127.8.39.190:36635/ using document root <none> and password file <none>
I20260618 23:30:38.002435 30471 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:38.002507 30471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:38.002662 30471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:38.004351 30471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-0/data/instance:
uuid: "770aa878b4f9416dba4314e3d78fd58b"
format_stamp: "Formatted at 2026-06-18 23:30:37 on dist-test-slave-47qt"
server_key: "d998b811d6c4d42f2128ed29ecd3c99b"
server_key_iv: "0c55bc2f6a0baaa16b9ecab54dcf2de3"
server_key_version: "encryptionkey@0"
I20260618 23:30:38.004791 30471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-0/wal/instance:
uuid: "770aa878b4f9416dba4314e3d78fd58b"
format_stamp: "Formatted at 2026-06-18 23:30:37 on dist-test-slave-47qt"
server_key: "d998b811d6c4d42f2128ed29ecd3c99b"
server_key_iv: "0c55bc2f6a0baaa16b9ecab54dcf2de3"
server_key_version: "encryptionkey@0"
I20260618 23:30:38.006299 30471 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260618 23:30:38.007107 30486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:38.007310 30471 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:30:38.007438 30471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-0/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-0/wal
uuid: "770aa878b4f9416dba4314e3d78fd58b"
format_stamp: "Formatted at 2026-06-18 23:30:37 on dist-test-slave-47qt"
server_key: "d998b811d6c4d42f2128ed29ecd3c99b"
server_key_iv: "0c55bc2f6a0baaa16b9ecab54dcf2de3"
server_key_version: "encryptionkey@0"
I20260618 23:30:38.007520 30471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:38.022004 30471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:38.024964 30471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:38.025296 30471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:38.030689 30471 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.190:44193
I20260618 23:30:38.030725 30538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.190:44193 every 8 connection(s)
I20260618 23:30:38.031080 30471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-0/data/info.pb
I20260618 23:30:38.032328 30539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:38.034047 30539 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44193 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:30:38.040474 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 30471
I20260618 23:30:38.040576 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-0/wal/instance
I20260618 23:30:38.040807 8350 external_mini_cluster.cc:1428] Setting key f3b2923bfceefe050b02c703c6f9e3b1
Jun 18 23:30:38 dist-test-slave-47qt krb5kdc[30458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825437, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:30:38.045697 30542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.034873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38818 (local address 127.8.39.190:44193)
0618 23:30:38.035036 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:38.035045 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:38.035345 (+ 300us) server_negotiation.cc:407] Connection header received
0618 23:30:38.036111 (+ 766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:38.036114 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:38.036135 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:38.036229 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:38.036877 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.038015 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.038899 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.039221 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.039323 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:38.042095 (+ 2772us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:38.042102 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:38.042103 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:38.042112 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:38.043929 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:38.044468 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:38.044471 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:38.044471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:38.044551 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:38.044825 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:38.044825 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:38.044826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:38.045040 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:38.045102 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:38.045439 (+ 337us) server_negotiation.cc:299] Negotiation successful
0618 23:30:38.045537 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":94,"threads_started":1}
I20260618 23:30:38.045854 30541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.034765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:44193 (local address 127.0.0.1:38818)
0618 23:30:38.034969 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:38.035147 (+ 178us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:38.035917 (+ 770us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:38.036308 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:38.036310 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:38.036560 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:38.036715 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:38.036719 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.038131 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.038132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:38.038796 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:38.038797 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.039379 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.039380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:38.039473 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:38.040152 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:30:38.040160 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:38.041979 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:38.044023 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:38.044031 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:38.044033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:38.044363 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:38.044630 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:38.044631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:38.044632 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:38.044747 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:38.045146 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:38.045154 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:38.045241 (+ 87us) client_negotiation.cc:769] Sending connection context
0618 23:30:38.045385 (+ 144us) client_negotiation.cc:240] Negotiation successful
0618 23:30:38.045656 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":77,"thread_start_us":110,"threads_started":1}
I20260618 23:30:38.046715 30539 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.39.189" port: 38543 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:30:38.047089 30541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.046949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:38543 (local address 127.0.0.1:53202)
0618 23:30:38.047005 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:38.047059 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.189:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260618 23:30:38.047219 30489 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.39.189:38543: connect: Connection refused (error 111)
I20260618 23:30:38.047663 30541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.047573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:38543 (local address 127.0.0.1:53212)
0618 23:30:38.047621 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:38.047645 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.189:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260618 23:30:38.047803 30539 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.39.189:38543: Network error: Client connection negotiation failed: client connection to 127.8.39.189:38543: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.8.39.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.189.keytab.
Entry for principal kudu/127.8.39.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.189.keytab.
I20260618 23:30:38.065811 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.189:38543
--webserver_interface=127.8.39.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.189.keytab
--principal=kudu/127.8.39.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:42493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:44193,127.8.39.189:38543,127.8.39.188:42657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:30:38.097060 30539 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.39.189" port: 38543 } attempt: 1
I20260618 23:30:38.097637 30541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.097503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:38543 (local address 127.0.0.1:53222)
0618 23:30:38.097568 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:38.097608 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.189:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260618 23:30:38.098022 30541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.097939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:38543 (local address 127.0.0.1:53236)
0618 23:30:38.097982 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:38.098006 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.189:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260618 23:30:38.098129 30539 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.39.189:38543: Network error: Client connection negotiation failed: client connection to 127.8.39.189:38543: connect: Connection refused (error 111)
W20260618 23:30:38.147369 30546 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260618 23:30:38.147595 30546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:38.147612 30546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:38.147634 30546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:38.148909 30546 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:30:38.149036 30546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:38.149086 30546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:38.149168 30546 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:30:38.149202 30546 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:30:38.150789 30546 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:42493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:44193,127.8.39.189:38543,127.8.39.188:42657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.189.keytab
--principal=kudu/127.8.39.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.189:38543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.39.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30546
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:38.150982 30546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:38.151152 30546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:38.156757 30552 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:38.156757 30554 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:38.156785 30551 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:38.157178 30546 server_base.cc:1034] running on GCE node
I20260618 23:30:38.157500 30546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:38.157876 30546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:38.159049 30546 hybrid_clock.cc:648] HybridClock initialized: now 1781825438159015 us; error 54 us; skew 500 ppm
Jun 18 23:30:38 dist-test-slave-47qt krb5kdc[30458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825438, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:38.162382 30546 init.cc:377] Logged in from keytab as kudu/127.8.39.189@KRBTEST.COM (short username kudu)
I20260618 23:30:38.162834 30546 webserver.cc:460] Webserver started at http://127.8.39.189:35219/ using document root <none> and password file <none>
I20260618 23:30:38.163079 30546 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:38.163151 30546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:38.163304 30546 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:38.164350 30546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-1/data/instance:
uuid: "2aca02174f9e4433bfea288882e1e693"
format_stamp: "Formatted at 2026-06-18 23:30:38 on dist-test-slave-47qt"
server_key: "24b556d95d6ba230025aa33e7cfb56d5"
server_key_iv: "1606ea199b8b8b8411973c857ab1cfb6"
server_key_version: "encryptionkey@0"
I20260618 23:30:38.164695 30546 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-1/wal/instance:
uuid: "2aca02174f9e4433bfea288882e1e693"
format_stamp: "Formatted at 2026-06-18 23:30:38 on dist-test-slave-47qt"
server_key: "24b556d95d6ba230025aa33e7cfb56d5"
server_key_iv: "1606ea199b8b8b8411973c857ab1cfb6"
server_key_version: "encryptionkey@0"
I20260618 23:30:38.166096 30546 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260618 23:30:38.166326 30539 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.39.189" port: 38543 } attempt: 2
I20260618 23:30:38.166801 30541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.166664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:38543 (local address 127.0.0.1:53240)
0618 23:30:38.166728 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:38.166765 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.189:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260618 23:30:38.167596 30541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.167350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:38543 (local address 127.0.0.1:53244)
0618 23:30:38.167529 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:38.167562 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.189:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
W20260618 23:30:38.167744 30539 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.39.189:38543: Network error: Client connection negotiation failed: client connection to 127.8.39.189:38543: connect: Connection refused (error 111)
I20260618 23:30:38.167755 30561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:38.168030 30546 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260618 23:30:38.168130 30546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-1/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-1/wal
uuid: "2aca02174f9e4433bfea288882e1e693"
format_stamp: "Formatted at 2026-06-18 23:30:38 on dist-test-slave-47qt"
server_key: "24b556d95d6ba230025aa33e7cfb56d5"
server_key_iv: "1606ea199b8b8b8411973c857ab1cfb6"
server_key_version: "encryptionkey@0"
I20260618 23:30:38.168218 30546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:38.187268 30546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:38.189376 30546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:38.189546 30546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:38.193547 30546 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.189:38543
I20260618 23:30:38.193576 30613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.189:38543 every 8 connection(s)
I20260618 23:30:38.193928 30546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-1/data/info.pb
I20260618 23:30:38.194906 30614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:38.196521 30614 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44193 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:30:38.200834 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 30546
I20260618 23:30:38.200933 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-1/wal/instance
I20260618 23:30:38.201160 8350 external_mini_cluster.cc:1428] Setting key 0e9f7cf37741881a2870891456d17cff
Jun 18 23:30:38 dist-test-slave-47qt krb5kdc[30458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825438, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.189@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:30:38.206581 30542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.197280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38822 (local address 127.8.39.190:44193)
0618 23:30:38.197328 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:38.197330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:38.197588 (+ 258us) server_negotiation.cc:407] Connection header received
0618 23:30:38.198155 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:38.198156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:38.198169 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:38.198229 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:38.198897 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.199410 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.200373 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.200579 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.200680 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:38.203137 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:38.203138 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:38.203139 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:38.203146 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:38.204935 (+ 1789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:38.205467 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:38.205469 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:38.205469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:38.205536 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:38.205829 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:38.205830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:38.205830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:38.205990 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:38.206068 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:38.206334 (+ 266us) server_negotiation.cc:299] Negotiation successful
0618 23:30:38.206442 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260618 23:30:38.206599 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.197279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:44193 (local address 127.0.0.1:38822)
0618 23:30:38.197456 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:38.197491 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:38.197942 (+ 451us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:38.198350 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:38.198353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:38.198590 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:38.198794 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:38.198798 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.199496 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.199497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:38.200284 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:38.200286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.200699 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.200700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:38.200818 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:38.201434 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:30:38.201448 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:38.203025 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:38.205045 (+ 2020us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:38.205052 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:38.205054 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:38.205386 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:38.205639 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:38.205639 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:38.205640 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:38.205751 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:38.206070 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:38.206074 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:38.206199 (+ 125us) client_negotiation.cc:769] Sending connection context
0618 23:30:38.206296 (+ 97us) client_negotiation.cc:240] Negotiation successful
0618 23:30:38.206441 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260618 23:30:38.207161 30614 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.39.189" port: 38543 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.39.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.39.188@KRBTEST.COM" created.
Jun 18 23:30:38 dist-test-slave-47qt krb5kdc[30458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825438, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.189@KRBTEST.COM for kudu/127.8.39.189@KRBTEST.COM
I20260618 23:30:38.217725 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.207563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:38543 (local address 127.0.0.1:53252)
0618 23:30:38.207637 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:38.207649 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:38.207694 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:38.208474 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:38.208475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:38.208722 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:38.208790 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:38.208792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.209916 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.209917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:38.210622 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:38.210624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.211398 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.211400 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:38.211488 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:38.211944 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:30:38.211949 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:38.213832 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:38.216535 (+ 2703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:38.216536 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:38.216536 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:38.216712 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:38.216975 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:38.216975 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:38.216976 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:38.217036 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:38.217469 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:38.217470 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:38.217509 (+ 39us) client_negotiation.cc:769] Sending connection context
0618 23:30:38.217575 (+ 66us) client_negotiation.cc:240] Negotiation successful
0618 23:30:38.217658 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260618 23:30:38.217865 30618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.207666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53252 (local address 127.8.39.189:38543)
0618 23:30:38.208054 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:38.208116 (+ 62us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:38.208138 (+ 22us) server_negotiation.cc:407] Connection header received
0618 23:30:38.208158 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:38.208160 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:38.208173 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:38.208472 (+ 299us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:38.208897 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.209811 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.210716 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.211255 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.211399 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:38.213955 (+ 2556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:38.213968 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:38.213969 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:38.213983 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:38.216429 (+ 2446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:38.216803 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:38.216806 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:38.216807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:38.216875 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:38.217111 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:38.217112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:38.217113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:38.217375 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:38.217466 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:38.217633 (+ 167us) server_negotiation.cc:299] Negotiation successful
0618 23:30:38.217721 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":96,"threads_started":1}
I20260618 23:30:38.218528 30614 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.39.188" port: 42657 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:30:38.218904 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.218773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:42657 (local address 127.0.0.1:49604)
0618 23:30:38.218826 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:38.218873 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260618 23:30:38.219043 30562 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.39.188:42657: connect: Connection refused (error 111)
I20260618 23:30:38.219437 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.219348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:42657 (local address 127.0.0.1:49608)
0618 23:30:38.219394 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:38.219417 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260618 23:30:38.219540 30614 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.39.188:42657: Network error: Client connection negotiation failed: client connection to 127.8.39.188:42657: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.39.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.188.keytab.
Entry for principal kudu/127.8.39.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.188.keytab.
I20260618 23:30:38.227921 8350 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskNmhGNv/build/release/bin/kudu
/tmp/dist-test-taskNmhGNv/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.188:42657
--webserver_interface=127.8.39.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.188.keytab
--principal=kudu/127.8.39.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.39.148:42493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.39.190:44193,127.8.39.189:38543,127.8.39.188:42657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260618 23:30:38.258913 30539 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.39.189" port: 38543 } attempt: 3
Jun 18 23:30:38 dist-test-slave-47qt krb5kdc[30458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825437, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for kudu/127.8.39.189@KRBTEST.COM
I20260618 23:30:38.266518 30541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.259330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:38543 (local address 127.0.0.1:53264)
0618 23:30:38.259429 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:38.259448 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:38.259520 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:38.259681 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:38.259682 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:38.259848 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:38.259919 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:38.259920 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.260590 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.260591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:38.261167 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:38.261168 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.261585 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.261586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:38.261665 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:38.262121 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:30:38.262133 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:38.263346 (+ 1213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:38.265455 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:38.265456 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:38.265456 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:38.265624 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:38.265860 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:38.265861 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:38.265862 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:38.265924 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:38.266215 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:38.266216 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:38.266262 (+ 46us) client_negotiation.cc:769] Sending connection context
0618 23:30:38.266355 (+ 93us) client_negotiation.cc:240] Negotiation successful
0618 23:30:38.266438 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260618 23:30:38.266568 30618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.259437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53264 (local address 127.8.39.189:38543)
0618 23:30:38.259504 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:38.259507 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:38.259519 (+ 12us) server_negotiation.cc:407] Connection header received
0618 23:30:38.259588 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:38.259590 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:38.259602 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:38.259675 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:38.260003 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.260503 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.261268 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.261508 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.261588 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:38.263450 (+ 1862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:38.263452 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:38.263452 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:38.263460 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:38.265343 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:38.265699 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:38.265701 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:38.265701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:38.265765 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:38.265995 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:38.265995 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:38.265996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:38.266129 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:38.266196 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:38.266396 (+ 200us) server_negotiation.cc:299] Negotiation successful
0618 23:30:38.266476 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260618 23:30:38.266979 30539 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.39.188" port: 42657 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:30:38.267302 30541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.267207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:42657 (local address 127.0.0.1:49624)
0618 23:30:38.267242 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:38.267272 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260618 23:30:38.267589 30541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.267539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:42657 (local address 127.0.0.1:49630)
0618 23:30:38.267560 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:38.267578 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260618 23:30:38.267692 30539 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.39.188:42657: Network error: Client connection negotiation failed: client connection to 127.8.39.188:42657: connect: Connection refused (error 111)
I20260618 23:30:38.268908 30614 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.39.188" port: 42657 } attempt: 1
I20260618 23:30:38.269301 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.269176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:42657 (local address 127.0.0.1:49638)
0618 23:30:38.269228 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:38.269280 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260618 23:30:38.269747 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.269641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:42657 (local address 127.0.0.1:49644)
0618 23:30:38.269686 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:38.269711 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260618 23:30:38.269865 30614 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.39.188:42657: Network error: Client connection negotiation failed: client connection to 127.8.39.188:42657: connect: Connection refused (error 111)
I20260618 23:30:38.298884 30539 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.39.188" port: 42657 } attempt: 1
I20260618 23:30:38.299381 30541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.299239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:42657 (local address 127.0.0.1:49648)
0618 23:30:38.299304 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:38.299346 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260618 23:30:38.299826 30541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.299673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:42657 (local address 127.0.0.1:49656)
0618 23:30:38.299761 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:38.299804 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260618 23:30:38.300072 30539 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.39.188:42657: Network error: Client connection negotiation failed: client connection to 127.8.39.188:42657: connect: Connection refused (error 111)
W20260618 23:30:38.312876 30621 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260618 23:30:38.313088 30621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260618 23:30:38.313107 30621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260618 23:30:38.313191 30621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260618 23:30:38.314486 30621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260618 23:30:38.314540 30621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260618 23:30:38.314555 30621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260618 23:30:38.314568 30621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260618 23:30:38.314580 30621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260618 23:30:38.316028 30621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.39.148:42493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.39.190:44193,127.8.39.189:38543,127.8.39.188:42657
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/krb5kdc/kudu_127.8.39.188.keytab
--principal=kudu/127.8.39.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.39.188:42657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.39.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 18 Jun 2026 22:23:54 UTC on ccycloud.quasar-uzckeu.root.comops.site
build id 4454735
I20260618 23:30:38.316222 30621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260618 23:30:38.316390 30621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260618 23:30:38.321108 30627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260618 23:30:38.321117 30629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:38.321313 30621 server_base.cc:1034] running on GCE node
W20260618 23:30:38.321110 30626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260618 23:30:38.321620 30621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260618 23:30:38.321907 30621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260618 23:30:38.323071 30621 hybrid_clock.cc:648] HybridClock initialized: now 1781825438323058 us; error 45 us; skew 500 ppm
Jun 18 23:30:38 dist-test-slave-47qt krb5kdc[30458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825438, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260618 23:30:38.325443 30621 init.cc:377] Logged in from keytab as kudu/127.8.39.188@KRBTEST.COM (short username kudu)
I20260618 23:30:38.325897 30621 webserver.cc:460] Webserver started at http://127.8.39.188:44181/ using document root <none> and password file <none>
I20260618 23:30:38.326113 30621 fs_manager.cc:359] Metadata directory not provided
I20260618 23:30:38.326215 30621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260618 23:30:38.326368 30621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260618 23:30:38.327414 30621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-2/data/instance:
uuid: "48dd0cb815984105abe9300422576a56"
format_stamp: "Formatted at 2026-06-18 23:30:38 on dist-test-slave-47qt"
server_key: "7960fdf82d71ce4dd3e17a232f0bd396"
server_key_iv: "74fcd08be3cfe8650586fdef62b6ce18"
server_key_version: "encryptionkey@0"
I20260618 23:30:38.327771 30621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-2/wal/instance:
uuid: "48dd0cb815984105abe9300422576a56"
format_stamp: "Formatted at 2026-06-18 23:30:38 on dist-test-slave-47qt"
server_key: "7960fdf82d71ce4dd3e17a232f0bd396"
server_key_iv: "74fcd08be3cfe8650586fdef62b6ce18"
server_key_version: "encryptionkey@0"
I20260618 23:30:38.329335 30621 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260618 23:30:38.330106 30636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260618 23:30:38.330498 30621 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260618 23:30:38.330622 30621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-2/data,/tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-2/wal
uuid: "48dd0cb815984105abe9300422576a56"
format_stamp: "Formatted at 2026-06-18 23:30:38 on dist-test-slave-47qt"
server_key: "7960fdf82d71ce4dd3e17a232f0bd396"
server_key_iv: "74fcd08be3cfe8650586fdef62b6ce18"
server_key_version: "encryptionkey@0"
I20260618 23:30:38.330711 30621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260618 23:30:38.338038 30614 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.39.188" port: 42657 } attempt: 2
I20260618 23:30:38.338477 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.338351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:42657 (local address 127.0.0.1:49664)
0618 23:30:38.338414 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:38.338452 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260618 23:30:38.338804 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.338697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:42657 (local address 127.0.0.1:49668)
0618 23:30:38.338755 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:38.338778 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.39.188:42657: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260618 23:30:38.338884 30614 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.39.188:42657: Network error: Client connection negotiation failed: client connection to 127.8.39.188:42657: connect: Connection refused (error 111)
I20260618 23:30:38.367396 30621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260618 23:30:38.369757 30621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260618 23:30:38.369928 30621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260618 23:30:38.374075 30621 rpc_server.cc:307] RPC server started. Bound to: 127.8.39.188:42657
I20260618 23:30:38.374154 30688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.39.188:42657 every 8 connection(s)
I20260618 23:30:38.374508 30621 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-2/data/info.pb
I20260618 23:30:38.375257 30539 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.39.188" port: 42657 } attempt: 2
I20260618 23:30:38.375542 30689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260618 23:30:38.377671 30689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44193 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 23:30:38 dist-test-slave-47qt krb5kdc[30458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825437, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.190@KRBTEST.COM for kudu/127.8.39.188@KRBTEST.COM
Jun 18 23:30:38 dist-test-slave-47qt krb5kdc[30458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825438, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.188@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:30:38.383164 8350 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskNmhGNv/build/release/bin/kudu as pid 30621
I20260618 23:30:38.383280 8350 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskNmhGNv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781825053542692-8350-0/minicluster-data/master-2/wal/instance
I20260618 23:30:38.383519 8350 external_mini_cluster.cc:1428] Setting key 534ad7d2075be467f9cb50090521f9bc
I20260618 23:30:38.386174 30541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.375623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:42657 (local address 127.0.0.1:49684)
0618 23:30:38.375688 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:38.375698 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:38.375858 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:38.377172 (+ 1314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:38.377173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:38.377364 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:38.377409 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:38.377411 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.378682 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.378683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:38.379400 (+ 717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:38.379402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.380097 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.380099 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:38.380281 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:38.380775 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:30:38.380779 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:38.382439 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:38.384654 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:38.384655 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:38.384655 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:38.384897 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:38.385203 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:38.385204 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:38.385205 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:38.385277 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:38.385794 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:38.385795 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:38.385844 (+ 49us) client_negotiation.cc:769] Sending connection context
0618 23:30:38.385974 (+ 130us) client_negotiation.cc:240] Negotiation successful
0618 23:30:38.386070 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260618 23:30:38.386788 30691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.375774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49684 (local address 127.8.39.188:42657)
0618 23:30:38.376050 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:38.376062 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:38.376110 (+ 48us) server_negotiation.cc:407] Connection header received
0618 23:30:38.376895 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:38.376905 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:38.376961 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:38.377187 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:38.377526 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.378572 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.379553 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.379964 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.380117 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:38.382541 (+ 2424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:38.382549 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:38.382549 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:38.382555 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:38.384561 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:38.385010 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:38.385013 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:38.385013 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:38.385138 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:38.385355 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:38.385356 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:38.385356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:38.385689 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:38.385765 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:38.386434 (+ 669us) server_negotiation.cc:299] Negotiation successful
0618 23:30:38.386571 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":97,"threads_started":1}
I20260618 23:30:38.386941 30692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.378334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.190:44193 (local address 127.0.0.1:38838)
0618 23:30:38.378528 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:38.378568 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:38.378642 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:38.378809 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:38.378811 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:38.379049 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:38.379108 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:38.379109 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.380106 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.380107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:38.380914 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:38.380916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.381511 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.381512 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:38.381595 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:38.382071 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:30:38.382081 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:38.383648 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:38.385647 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:38.385654 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:38.385656 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:38.385948 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:38.386167 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:38.386167 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:38.386168 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:38.386230 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:38.386517 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:38.386524 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:38.386579 (+ 55us) client_negotiation.cc:769] Sending connection context
0618 23:30:38.386663 (+ 84us) client_negotiation.cc:240] Negotiation successful
0618 23:30:38.386779 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":117,"threads_started":1}
I20260618 23:30:38.387214 30542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.378379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38838 (local address 127.8.39.190:44193)
0618 23:30:38.378444 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:38.378447 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:38.378692 (+ 245us) server_negotiation.cc:407] Connection header received
0618 23:30:38.378713 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:38.378714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:38.378724 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:38.378780 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:38.379180 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.379925 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.381034 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.381367 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.381443 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:38.383762 (+ 2319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:38.383764 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:38.383764 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:38.383771 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:38.385562 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:38.386031 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:38.386033 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:38.386033 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:38.386088 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:38.386315 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:38.386316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:38.386316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:38.386433 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:38.386473 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:38.386704 (+ 231us) server_negotiation.cc:299] Negotiation successful
0618 23:30:38.386777 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":39}
I20260618 23:30:38.387603 30689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.39.189" port: 38543 } has no permanent_uuid. Determining permanent_uuid...
I20260618 23:30:38.389256 30539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 770aa878b4f9416dba4314e3d78fd58b: Bootstrap starting.
I20260618 23:30:38.390061 30539 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 770aa878b4f9416dba4314e3d78fd58b: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:38.390365 30539 log.cc:824] T 00000000000000000000000000000000 P 770aa878b4f9416dba4314e3d78fd58b: Log is configured to *not* fsync() on all Append() calls
Jun 18 23:30:38 dist-test-slave-47qt krb5kdc[30458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.188@KRBTEST.COM
I20260618 23:30:38.391186 30539 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 770aa878b4f9416dba4314e3d78fd58b: No bootstrap required, opened a new log
Jun 18 23:30:38 dist-test-slave-47qt krb5kdc[30458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825438, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.188@KRBTEST.COM for kudu/127.8.39.189@KRBTEST.COM
I20260618 23:30:38.394239 30694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.384243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49688 (local address 127.8.39.188:42657)
0618 23:30:38.384704 (+ 461us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:38.384706 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:38.384711 (+ 5us) server_negotiation.cc:407] Connection header received
0618 23:30:38.384725 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:38.384726 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:38.384734 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:38.384803 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:38.385678 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.386383 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.387901 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.388148 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.388219 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:38.391077 (+ 2858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:38.391078 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:38.391079 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:38.391086 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:38.392954 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:38.393381 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:38.393382 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:38.393383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:38.393453 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:38.393666 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:38.393667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:38.393667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:38.393792 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:38.393940 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:38.394013 (+ 73us) server_negotiation.cc:299] Negotiation successful
0618 23:30:38.394091 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":184,"threads_started":1}
I20260618 23:30:38.394297 30539 raft_consensus.cc:348] T 00000000000000000000000000000000 P 770aa878b4f9416dba4314e3d78fd58b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770aa878b4f9416dba4314e3d78fd58b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44193 } } peers { permanent_uuid: "2aca02174f9e4433bfea288882e1e693" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 38543 } } peers { permanent_uuid: "48dd0cb815984105abe9300422576a56" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 42657 } }
I20260618 23:30:38.394459 30539 raft_consensus.cc:729] T 00000000000000000000000000000000 P 770aa878b4f9416dba4314e3d78fd58b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 770aa878b4f9416dba4314e3d78fd58b, State: Initialized, Role: FOLLOWER
I20260618 23:30:38.394615 30539 consensus_queue.cc:260] T 00000000000000000000000000000000 P 770aa878b4f9416dba4314e3d78fd58b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770aa878b4f9416dba4314e3d78fd58b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44193 } } peers { permanent_uuid: "2aca02174f9e4433bfea288882e1e693" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 38543 } } peers { permanent_uuid: "48dd0cb815984105abe9300422576a56" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 42657 } }
I20260618 23:30:38.395022 30539 sys_catalog.cc:564] T 00000000000000000000000000000000 P 770aa878b4f9416dba4314e3d78fd58b [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:30:38.396541 30696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 770aa878b4f9416dba4314e3d78fd58b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770aa878b4f9416dba4314e3d78fd58b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44193 } } peers { permanent_uuid: "2aca02174f9e4433bfea288882e1e693" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 38543 } } peers { permanent_uuid: "48dd0cb815984105abe9300422576a56" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 42657 } } }
I20260618 23:30:38.396864 30696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 770aa878b4f9416dba4314e3d78fd58b [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:30:38.397467 30618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.388016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53272 (local address 127.8.39.189:38543)
0618 23:30:38.388058 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:38.388060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:38.388067 (+ 7us) server_negotiation.cc:407] Connection header received
0618 23:30:38.388092 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:38.388098 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:38.388107 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:38.388169 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:38.388816 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.389569 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.390784 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.391052 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.391129 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:38.394132 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:38.394134 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:38.394134 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:38.394142 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:38.395943 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:38.396390 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:38.396391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:38.396391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:38.396455 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:38.396786 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:38.396787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:38.396787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:38.396925 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:38.396979 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:38.397217 (+ 238us) server_negotiation.cc:299] Negotiation successful
0618 23:30:38.397315 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260618 23:30:38.397672 30707 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 770aa878b4f9416dba4314e3d78fd58b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 23:30:38.397723 30707 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:30:38.398033 30692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.387908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.189:38543 (local address 127.0.0.1:53272)
0618 23:30:38.387942 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:38.387954 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:38.387988 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:38.388189 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:38.388190 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:38.388411 (+ 221us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:38.388470 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:38.388472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.389813 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.389814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:38.390674 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:38.390677 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.391125 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.391126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:38.391208 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:38.392498 (+ 1290us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:30:38.392503 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:38.394021 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:38.396043 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:38.396044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:38.396044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:38.396294 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:38.396637 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:38.396637 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:38.396638 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:38.396706 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:38.397029 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:38.397030 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:38.397076 (+ 46us) client_negotiation.cc:769] Sending connection context
0618 23:30:38.397581 (+ 505us) client_negotiation.cc:240] Negotiation successful
0618 23:30:38.397930 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260618 23:30:38.398396 30689 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.39.188" port: 42657 } has no permanent_uuid. Determining permanent_uuid...
Jun 18 23:30:38 dist-test-slave-47qt krb5kdc[30458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.190@KRBTEST.COM
I20260618 23:30:38.397989 30471 master_runner.cc:186] Error getting master registration for 127.8.39.189:38543: OK, instance_id { permanent_uuid: "2aca02174f9e4433bfea288882e1e693" instance_seqno: 1781825438192364 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260618 23:30:38.401031 30471 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260618 23:30:38.403280 30542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.394877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38842 (local address 127.8.39.190:44193)
0618 23:30:38.394920 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:38.394922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:38.394962 (+ 40us) server_negotiation.cc:407] Connection header received
0618 23:30:38.395032 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:38.395032 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:38.395042 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:38.395094 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:38.395613 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.396161 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.397104 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.397412 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.397477 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:38.400144 (+ 2667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:38.400146 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:38.400146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:38.400153 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:38.401872 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:38.402204 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:38.402205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:38.402206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:38.402266 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:38.402475 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:38.402476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:38.402476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:38.402617 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:38.402748 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:38.403114 (+ 366us) server_negotiation.cc:299] Negotiation successful
0618 23:30:38.403182 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
Jun 18 23:30:38 dist-test-slave-47qt krb5kdc[30458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825438, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.188@KRBTEST.COM for kudu/127.8.39.188@KRBTEST.COM
Jun 18 23:30:38 dist-test-slave-47qt krb5kdc[30458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825437, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.39.189@KRBTEST.COM
I20260618 23:30:38.409716 30692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.401249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:42657 (local address 127.0.0.1:49690)
0618 23:30:38.401300 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:38.401310 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:38.401349 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:38.401485 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:38.401486 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:38.401626 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:38.401670 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:38.401672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.402616 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.402617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:38.403376 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:38.403378 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.403778 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.403779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:38.403826 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:38.404222 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:30:38.404227 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:38.405879 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:38.408370 (+ 2491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:38.408371 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:38.408371 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:38.408595 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:38.408820 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:38.408821 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:38.408821 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:38.408896 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:38.409236 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:38.409237 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:38.409332 (+ 95us) client_negotiation.cc:769] Sending connection context
0618 23:30:38.409456 (+ 124us) client_negotiation.cc:240] Negotiation successful
0618 23:30:38.409558 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260618 23:30:38.409897 30694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.401301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49690 (local address 127.8.39.188:42657)
0618 23:30:38.401339 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:38.401341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:38.401350 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:30:38.401401 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:38.401402 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:38.401413 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:38.401484 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:38.401755 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.402526 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.403476 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.403704 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.403784 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:38.405972 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:38.405973 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:38.405974 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:38.405981 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:38.408278 (+ 2297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:38.408679 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:38.408680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:38.408681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:38.408744 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:38.408959 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:38.408960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:38.408961 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:38.409100 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:38.409214 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:38.409718 (+ 504us) server_negotiation.cc:299] Negotiation successful
0618 23:30:38.409815 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260618 23:30:38.411646 30689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48dd0cb815984105abe9300422576a56: Bootstrap starting.
I20260618 23:30:38.412334 30689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48dd0cb815984105abe9300422576a56: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:38.412405 30618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.403837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53280 (local address 127.8.39.189:38543)
0618 23:30:38.403883 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:38.403885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:38.404188 (+ 303us) server_negotiation.cc:407] Connection header received
0618 23:30:38.404213 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:38.404214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:38.404223 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:38.404281 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:38.404893 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.405627 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.406438 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.406749 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.406825 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:38.409219 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:38.409220 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:38.409220 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:38.409227 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:38.410967 (+ 1740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:38.411337 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:38.411337 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:38.411338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:38.411395 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:38.411603 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:38.411604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:38.411604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:38.411735 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:38.411867 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:38.412245 (+ 378us) server_negotiation.cc:299] Negotiation successful
0618 23:30:38.412316 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260618 23:30:38.412672 30689 log.cc:824] T 00000000000000000000000000000000 P 48dd0cb815984105abe9300422576a56: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:38.413620 30689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48dd0cb815984105abe9300422576a56: No bootstrap required, opened a new log
I20260618 23:30:38.413868 8350 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260618 23:30:38.414851 30513 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "770aa878b4f9416dba4314e3d78fd58b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38842
I20260618 23:30:38.415006 30513 raft_consensus.cc:482] T 00000000000000000000000000000000 P 770aa878b4f9416dba4314e3d78fd58b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260618 23:30:38.415055 30513 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 770aa878b4f9416dba4314e3d78fd58b [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:38.415462 30689 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48dd0cb815984105abe9300422576a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770aa878b4f9416dba4314e3d78fd58b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44193 } } peers { permanent_uuid: "2aca02174f9e4433bfea288882e1e693" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 38543 } } peers { permanent_uuid: "48dd0cb815984105abe9300422576a56" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 42657 } }
I20260618 23:30:38.415606 30689 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48dd0cb815984105abe9300422576a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48dd0cb815984105abe9300422576a56, State: Initialized, Role: FOLLOWER
I20260618 23:30:38.415692 30689 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48dd0cb815984105abe9300422576a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770aa878b4f9416dba4314e3d78fd58b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44193 } } peers { permanent_uuid: "2aca02174f9e4433bfea288882e1e693" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 38543 } } peers { permanent_uuid: "48dd0cb815984105abe9300422576a56" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 42657 } }
I20260618 23:30:38.415894 30513 raft_consensus.cc:504] T 00000000000000000000000000000000 P 770aa878b4f9416dba4314e3d78fd58b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770aa878b4f9416dba4314e3d78fd58b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44193 } } peers { permanent_uuid: "2aca02174f9e4433bfea288882e1e693" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 38543 } } peers { permanent_uuid: "48dd0cb815984105abe9300422576a56" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 42657 } }
I20260618 23:30:38.416011 30689 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48dd0cb815984105abe9300422576a56 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:30:38.416134 30513 leader_election.cc:288] T 00000000000000000000000000000000 P 770aa878b4f9416dba4314e3d78fd58b [CANDIDATE]: Term 1 election: Requested vote from peers 2aca02174f9e4433bfea288882e1e693 (127.8.39.189:38543), 48dd0cb815984105abe9300422576a56 (127.8.39.188:42657)
I20260618 23:30:38.416563 30663 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "770aa878b4f9416dba4314e3d78fd58b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "48dd0cb815984105abe9300422576a56"
I20260618 23:30:38.416739 30663 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48dd0cb815984105abe9300422576a56 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:38.416042 30710 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48dd0cb815984105abe9300422576a56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770aa878b4f9416dba4314e3d78fd58b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44193 } } peers { permanent_uuid: "2aca02174f9e4433bfea288882e1e693" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 38543 } } peers { permanent_uuid: "48dd0cb815984105abe9300422576a56" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 42657 } } }
I20260618 23:30:38.417878 30710 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48dd0cb815984105abe9300422576a56 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:30:38.418165 30663 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 48dd0cb815984105abe9300422576a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 770aa878b4f9416dba4314e3d78fd58b in term 1.
I20260618 23:30:38.418397 30487 leader_election.cc:302] T 00000000000000000000000000000000 P 770aa878b4f9416dba4314e3d78fd58b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48dd0cb815984105abe9300422576a56, 770aa878b4f9416dba4314e3d78fd58b; no voters:
I20260618 23:30:38.418568 30696 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 770aa878b4f9416dba4314e3d78fd58b [term 1 FOLLOWER]: Leader election won for term 1
I20260618 23:30:38.418712 30696 raft_consensus.cc:686] T 00000000000000000000000000000000 P 770aa878b4f9416dba4314e3d78fd58b [term 1 LEADER]: Becoming Leader. State: Replica: 770aa878b4f9416dba4314e3d78fd58b, State: Running, Role: LEADER
I20260618 23:30:38.418807 30696 consensus_queue.cc:237] T 00000000000000000000000000000000 P 770aa878b4f9416dba4314e3d78fd58b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770aa878b4f9416dba4314e3d78fd58b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44193 } } peers { permanent_uuid: "2aca02174f9e4433bfea288882e1e693" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 38543 } } peers { permanent_uuid: "48dd0cb815984105abe9300422576a56" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 42657 } }
I20260618 23:30:38.419273 30696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 770aa878b4f9416dba4314e3d78fd58b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 770aa878b4f9416dba4314e3d78fd58b. Latest consensus state: current_term: 1 leader_uuid: "770aa878b4f9416dba4314e3d78fd58b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770aa878b4f9416dba4314e3d78fd58b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44193 } } peers { permanent_uuid: "2aca02174f9e4433bfea288882e1e693" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 38543 } } peers { permanent_uuid: "48dd0cb815984105abe9300422576a56" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 42657 } } }
I20260618 23:30:38.419353 30696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 770aa878b4f9416dba4314e3d78fd58b [sys.catalog]: This master's current role is: LEADER
W20260618 23:30:38.419750 30728 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 48dd0cb815984105abe9300422576a56: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260618 23:30:38.419807 30728 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260618 23:30:38.421990 30621 master_runner.cc:186] Error getting master registration for 127.8.39.189:38543: OK, instance_id { permanent_uuid: "2aca02174f9e4433bfea288882e1e693" instance_seqno: 1781825438192364 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260618 23:30:38.424206 30621 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260618 23:30:38.428119 30694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.417642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49706 (local address 127.8.39.188:42657)
0618 23:30:38.417700 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:38.417702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:38.420333 (+ 2631us) server_negotiation.cc:407] Connection header received
0618 23:30:38.420437 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:38.420438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:38.420448 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:38.420479 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:38.421235 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.421912 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.422927 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.423211 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.423253 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:38.424873 (+ 1620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:38.424875 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:38.424875 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:38.424882 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:38.426858 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:38.427261 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:38.427262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:38.427263 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:38.427317 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:38.427536 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:38.427536 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:38.427537 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:38.427657 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:38.427727 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:38.427941 (+ 214us) server_negotiation.cc:299] Negotiation successful
0618 23:30:38.428021 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260618 23:30:38.429706 30730 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260618 23:30:38.429855 30730 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260618 23:30:38.430037 30614 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.39.188" port: 42657 } attempt: 3
I20260618 23:30:38.431384 30663 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 48dd0cb815984105abe9300422576a56 [term 1 FOLLOWER]: Refusing update from remote peer 770aa878b4f9416dba4314e3d78fd58b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:30:38.431571 30618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.417498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53296 (local address 127.8.39.189:38543)
0618 23:30:38.417591 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:38.417594 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:38.418065 (+ 471us) server_negotiation.cc:407] Connection header received
0618 23:30:38.418151 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:38.418152 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:38.418164 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:38.418226 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:38.419976 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.420708 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.425775 (+ 5067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.426088 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.426128 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:38.428047 (+ 1919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:38.428048 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:38.428048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:38.428066 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:38.429785 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:38.430482 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:38.430483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:38.430484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:38.430550 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:38.430901 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:38.430902 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:38.430902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:38.431038 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:38.431102 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:38.431393 (+ 291us) server_negotiation.cc:299] Negotiation successful
0618 23:30:38.431482 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260618 23:30:38.431687 30696 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 770aa878b4f9416dba4314e3d78fd58b [LEADER]: Connected to new peer: Peer: permanent_uuid: "48dd0cb815984105abe9300422576a56" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 42657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:30:38.434249 30696 sys_catalog.cc:455] T 00000000000000000000000000000000 P 770aa878b4f9416dba4314e3d78fd58b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "770aa878b4f9416dba4314e3d78fd58b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770aa878b4f9416dba4314e3d78fd58b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44193 } } peers { permanent_uuid: "2aca02174f9e4433bfea288882e1e693" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 38543 } } peers { permanent_uuid: "48dd0cb815984105abe9300422576a56" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 42657 } } }
I20260618 23:30:38.434346 30696 sys_catalog.cc:458] T 00000000000000000000000000000000 P 770aa878b4f9416dba4314e3d78fd58b [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:38.434883 30738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48dd0cb815984105abe9300422576a56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 770aa878b4f9416dba4314e3d78fd58b. Latest consensus state: current_term: 1 leader_uuid: "770aa878b4f9416dba4314e3d78fd58b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770aa878b4f9416dba4314e3d78fd58b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44193 } } peers { permanent_uuid: "2aca02174f9e4433bfea288882e1e693" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 38543 } } peers { permanent_uuid: "48dd0cb815984105abe9300422576a56" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 42657 } } }
I20260618 23:30:38.435005 30730 catalog_manager.cc:1347] Generated new cluster ID: d7701a0e7e41458b999c32ddf734fe23
I20260618 23:30:38.435046 30730 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260618 23:30:38.435137 30738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48dd0cb815984105abe9300422576a56 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:30:38.435290 30738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48dd0cb815984105abe9300422576a56 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "770aa878b4f9416dba4314e3d78fd58b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770aa878b4f9416dba4314e3d78fd58b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44193 } } peers { permanent_uuid: "2aca02174f9e4433bfea288882e1e693" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 38543 } } peers { permanent_uuid: "48dd0cb815984105abe9300422576a56" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 42657 } } }
I20260618 23:30:38.435447 30738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48dd0cb815984105abe9300422576a56 [sys.catalog]: This master's current role is: FOLLOWER
Jun 18 23:30:38 dist-test-slave-47qt krb5kdc[30458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781825438, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.39.189@KRBTEST.COM for kudu/127.8.39.188@KRBTEST.COM
I20260618 23:30:38.438735 30694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.430479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49712 (local address 127.8.39.188:42657)
0618 23:30:38.431259 (+ 780us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:38.431261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:38.431269 (+ 8us) server_negotiation.cc:407] Connection header received
0618 23:30:38.431286 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:38.431286 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:38.431294 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:38.431337 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:38.431737 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.432366 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.433674 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.433976 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.434045 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:38.436112 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:38.436113 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:38.436113 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:38.436119 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:38.437638 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:38.438008 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:38.438009 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:38.438009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:38.438073 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:38.438258 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:38.438259 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:38.438260 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:38.438370 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:38.438498 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:38.438579 (+ 81us) server_negotiation.cc:299] Negotiation successful
0618 23:30:38.438650 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760}
I20260618 23:30:38.438745 30616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.430263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.39.188:42657 (local address 127.0.0.1:49712)
0618 23:30:38.430353 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0618 23:30:38.430362 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0618 23:30:38.430398 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0618 23:30:38.431377 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0618 23:30:38.431378 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0618 23:30:38.431581 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0618 23:30:38.431622 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:38.431624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.432584 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.432585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:38.433412 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0618 23:30:38.433414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.434093 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.434094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0618 23:30:38.434147 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:38.434622 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0618 23:30:38.434626 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0618 23:30:38.436032 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0618 23:30:38.437716 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:38.437717 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:38.437718 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:38.437939 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:38.438140 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0618 23:30:38.438140 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0618 23:30:38.438141 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0618 23:30:38.438198 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0618 23:30:38.438435 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0618 23:30:38.438435 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0618 23:30:38.438480 (+ 45us) client_negotiation.cc:769] Sending connection context
0618 23:30:38.438555 (+ 75us) client_negotiation.cc:240] Negotiation successful
0618 23:30:38.438647 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260618 23:30:38.439368 30733 mvcc.cc:205] Tried to move back new op lower bound from 7298356995812524032 to 7298356995764203520. Current Snapshot: MvccSnapshot[applied={T|T < 7298356995812524032 or (T in {7298356995812524032})}]
I20260618 23:30:38.440620 30730 catalog_manager.cc:1370] Generated new certificate authority record
I20260618 23:30:38.440728 30614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aca02174f9e4433bfea288882e1e693: Bootstrap starting.
I20260618 23:30:38.441298 30730 catalog_manager.cc:1504] Loading token signing keys...
I20260618 23:30:38.441491 30614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2aca02174f9e4433bfea288882e1e693: Neither blocks nor log segments found. Creating new log.
I20260618 23:30:38.441771 30614 log.cc:824] T 00000000000000000000000000000000 P 2aca02174f9e4433bfea288882e1e693: Log is configured to *not* fsync() on all Append() calls
I20260618 23:30:38.442433 30614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2aca02174f9e4433bfea288882e1e693: No bootstrap required, opened a new log
I20260618 23:30:38.444227 30614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2aca02174f9e4433bfea288882e1e693 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770aa878b4f9416dba4314e3d78fd58b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44193 } } peers { permanent_uuid: "2aca02174f9e4433bfea288882e1e693" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 38543 } } peers { permanent_uuid: "48dd0cb815984105abe9300422576a56" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 42657 } }
I20260618 23:30:38.444350 30614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2aca02174f9e4433bfea288882e1e693 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2aca02174f9e4433bfea288882e1e693, State: Initialized, Role: FOLLOWER
I20260618 23:30:38.444432 30614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2aca02174f9e4433bfea288882e1e693 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770aa878b4f9416dba4314e3d78fd58b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44193 } } peers { permanent_uuid: "2aca02174f9e4433bfea288882e1e693" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 38543 } } peers { permanent_uuid: "48dd0cb815984105abe9300422576a56" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 42657 } }
I20260618 23:30:38.444708 30614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2aca02174f9e4433bfea288882e1e693 [sys.catalog]: configured and running, proceeding with master startup.
I20260618 23:30:38.444954 30588 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "770aa878b4f9416dba4314e3d78fd58b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2aca02174f9e4433bfea288882e1e693"
I20260618 23:30:38.445513 30588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2aca02174f9e4433bfea288882e1e693 [term 0 FOLLOWER]: Advancing to term 1
I20260618 23:30:38.446449 30588 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2aca02174f9e4433bfea288882e1e693 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 770aa878b4f9416dba4314e3d78fd58b in term 1.
I20260618 23:30:38.446897 30587 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2aca02174f9e4433bfea288882e1e693 [term 1 FOLLOWER]: Refusing update from remote peer 770aa878b4f9416dba4314e3d78fd58b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260618 23:30:38.446910 30542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:38.420096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38848 (local address 127.8.39.190:44193)
0618 23:30:38.420134 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:38.420136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:38.429255 (+ 9119us) server_negotiation.cc:407] Connection header received
0618 23:30:38.429374 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:38.429376 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:38.429387 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:38.429417 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:38.430519 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.431170 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.442084 (+ 10914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:38.442383 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:38.442462 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:38.443917 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0618 23:30:38.443919 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0618 23:30:38.443919 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0618 23:30:38.443927 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0618 23:30:38.445527 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:38.445900 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:38.445901 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:38.445901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:38.445957 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0618 23:30:38.446141 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0618 23:30:38.446142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0618 23:30:38.446142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0618 23:30:38.446264 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0618 23:30:38.446386 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:38.446724 (+ 338us) server_negotiation.cc:299] Negotiation successful
0618 23:30:38.446823 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260618 23:30:38.447438 30722 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 770aa878b4f9416dba4314e3d78fd58b [LEADER]: Connected to new peer: Peer: permanent_uuid: "2aca02174f9e4433bfea288882e1e693" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 38543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260618 23:30:38.447451 30740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aca02174f9e4433bfea288882e1e693 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770aa878b4f9416dba4314e3d78fd58b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44193 } } peers { permanent_uuid: "2aca02174f9e4433bfea288882e1e693" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 38543 } } peers { permanent_uuid: "48dd0cb815984105abe9300422576a56" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 42657 } } }
I20260618 23:30:38.447638 30740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aca02174f9e4433bfea288882e1e693 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:30:38.448848 30752 mvcc.cc:205] Tried to move back new op lower bound from 7298356995812524032 to 7298356995764203520. Current Snapshot: MvccSnapshot[applied={T|T < 7298356995812524032}]
I20260618 23:30:38.449422 30740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aca02174f9e4433bfea288882e1e693 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 770aa878b4f9416dba4314e3d78fd58b. Latest consensus state: current_term: 1 leader_uuid: "770aa878b4f9416dba4314e3d78fd58b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770aa878b4f9416dba4314e3d78fd58b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44193 } } peers { permanent_uuid: "2aca02174f9e4433bfea288882e1e693" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 38543 } } peers { permanent_uuid: "48dd0cb815984105abe9300422576a56" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 42657 } } }
I20260618 23:30:38.449491 30740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aca02174f9e4433bfea288882e1e693 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:30:38.449522 30740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2aca02174f9e4433bfea288882e1e693 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "770aa878b4f9416dba4314e3d78fd58b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770aa878b4f9416dba4314e3d78fd58b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44193 } } peers { permanent_uuid: "2aca02174f9e4433bfea288882e1e693" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 38543 } } peers { permanent_uuid: "48dd0cb815984105abe9300422576a56" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 42657 } } }
I20260618 23:30:38.449565 30740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2aca02174f9e4433bfea288882e1e693 [sys.catalog]: This master's current role is: FOLLOWER
I20260618 23:30:38.453620 30734 mvcc.cc:205] Tried to move back new op lower bound from 7298356995812524032 to 7298356995764203520. Current Snapshot: MvccSnapshot[applied={T|T < 7298356995812524032}]
I20260618 23:30:38.455164 30751 catalog_manager.cc:1259] Loaded cluster ID: d7701a0e7e41458b999c32ddf734fe23
I20260618 23:30:38.455221 30751 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2aca02174f9e4433bfea288882e1e693: loading cluster ID for follower catalog manager: success
I20260618 23:30:38.456311 30751 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2aca02174f9e4433bfea288882e1e693: acquiring CA information for follower catalog manager: success
W20260618 23:30:38.456465 30751 catalog_manager.cc:873] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260618 23:30:38.458621 30722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 770aa878b4f9416dba4314e3d78fd58b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "770aa878b4f9416dba4314e3d78fd58b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "770aa878b4f9416dba4314e3d78fd58b" member_type: VOTER last_known_addr { host: "127.8.39.190" port: 44193 } } peers { permanent_uuid: "2aca02174f9e4433bfea288882e1e693" member_type: VOTER last_known_addr { host: "127.8.39.189" port: 38543 } } peers { permanent_uuid: "48dd0cb815984105abe9300422576a56" member_type: VOTER last_known_addr { host: "127.8.39.188" port: 42657 } } }
I20260618 23:30:38.458720 30722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 770aa878b4f9416dba4314e3d78fd58b [sys.catalog]: This master's current role is: LEADER
I20260618 23:30:38.461196 30730 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 770aa878b4f9416dba4314e3d78fd58b: Generated new TSK 0
I20260618 23:30:38.461372 30730 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260618 23:30:38.485029 30618 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:30:38.483433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53302 (local address 127.8.39.189:38543)
0618 23:30:38.483479 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:38.483481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:38.484379 (+ 898us) server_negotiation.cc:407] Connection header received
0618 23:30:38.484485 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:38.484487 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:38.484497 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:38.484527 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:38.484955 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53302: BlockingRecv error: recv got EOF from 127.0.0.1:53302 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260618 23:30:38.485807 30542 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:30:38.483805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38860 (local address 127.8.39.190:44193)
0618 23:30:38.483846 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:38.483848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:38.485299 (+ 1451us) server_negotiation.cc:407] Connection header received
0618 23:30:38.485395 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:38.485396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:38.485407 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:38.485436 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:38.485745 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38860: BlockingRecv error: recv got EOF from 127.0.0.1:38860 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260618 23:30:38.486506 30694 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:30:38.484111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49716 (local address 127.8.39.188:42657)
0618 23:30:38.484150 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:38.484151 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:38.486016 (+ 1865us) server_negotiation.cc:407] Connection header received
0618 23:30:38.486109 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:38.486110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:38.486120 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:38.486149 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:38.486448 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49716: BlockingRecv error: recv got EOF from 127.0.0.1:49716 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260618 23:30:38.493896 30618 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:30:38.489804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53310 (local address 127.8.39.189:38543)
0618 23:30:38.489841 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:38.489842 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:38.493307 (+ 3465us) server_negotiation.cc:407] Connection header received
0618 23:30:38.493440 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:38.493441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:38.493463 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:38.493492 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:38.493842 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53310: BlockingRecv error: recv got EOF from 127.0.0.1:53310 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260618 23:30:38.497025 30618 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:30:38.496247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53320 (local address 127.8.39.189:38543)
0618 23:30:38.496280 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:38.496282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:38.496535 (+ 253us) server_negotiation.cc:407] Connection header received
0618 23:30:38.496632 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:38.496633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:38.496643 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:38.496670 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:38.496977 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53320: BlockingRecv error: recv got EOF from 127.0.0.1:53320 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260618 23:30:38.500958 30618 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:30:38.500168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53322 (local address 127.8.39.189:38543)
0618 23:30:38.500203 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:38.500204 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:38.500455 (+ 251us) server_negotiation.cc:407] Connection header received
0618 23:30:38.500549 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:38.500550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:38.500560 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:38.500588 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:38.500911 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53322: BlockingRecv error: recv got EOF from 127.0.0.1:53322 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260618 23:30:38.506860 30618 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:30:38.506091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53326 (local address 127.8.39.189:38543)
0618 23:30:38.506125 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:38.506127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:38.506353 (+ 226us) server_negotiation.cc:407] Connection header received
0618 23:30:38.506446 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:38.506447 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:38.506458 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:38.506485 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:38.506814 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53326: BlockingRecv error: recv got EOF from 127.0.0.1:53326 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260618 23:30:38.516968 30618 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:30:38.516132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53336 (local address 127.8.39.189:38543)
0618 23:30:38.516171 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:38.516173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:38.516410 (+ 237us) server_negotiation.cc:407] Connection header received
0618 23:30:38.516509 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:38.516510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:38.516521 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:38.516549 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:38.516911 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53336: BlockingRecv error: recv got EOF from 127.0.0.1:53336 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260618 23:30:38.535392 30618 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:30:38.534352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53350 (local address 127.8.39.189:38543)
0618 23:30:38.534394 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:38.534397 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:38.534673 (+ 276us) server_negotiation.cc:407] Connection header received
0618 23:30:38.534781 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:38.534782 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:38.534793 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:38.534822 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:38.535337 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53350: BlockingRecv error: recv got EOF from 127.0.0.1:53350 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260618 23:30:38.569620 30618 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:30:38.568678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53360 (local address 127.8.39.189:38543)
0618 23:30:38.568723 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:38.568726 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:38.569006 (+ 280us) server_negotiation.cc:407] Connection header received
0618 23:30:38.569109 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:38.569111 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:38.569152 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:38.569183 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:38.569562 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53360: BlockingRecv error: recv got EOF from 127.0.0.1:53360 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260618 23:30:38.635825 30618 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:30:38.635100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53364 (local address 127.8.39.189:38543)
0618 23:30:38.635197 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:38.635199 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:38.635319 (+ 120us) server_negotiation.cc:407] Connection header received
0618 23:30:38.635351 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:38.635352 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:38.635365 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:38.635420 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:38.635779 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53364: BlockingRecv error: recv got EOF from 127.0.0.1:53364 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260618 23:30:38.766113 30618 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:30:38.765155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53366 (local address 127.8.39.189:38543)
0618 23:30:38.765236 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:38.765239 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:38.765513 (+ 274us) server_negotiation.cc:407] Connection header received
0618 23:30:38.765630 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:38.765631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:38.765644 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:38.765719 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:38.766051 (+ 332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53366: BlockingRecv error: recv got EOF from 127.0.0.1:53366 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260618 23:30:39.024099 30618 negotiation.cc:337] Failed RPC negotiation. Trace:
0618 23:30:39.023425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53378 (local address 127.8.39.189:38543)
0618 23:30:39.023529 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:39.023531 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:39.023541 (+ 10us) server_negotiation.cc:407] Connection header received
0618 23:30:39.023577 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:39.023578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:39.023589 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:39.023664 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0618 23:30:39.024049 (+ 385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53378: BlockingRecv error: recv got EOF from 127.0.0.1:53378 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260618 23:30:39.420244 30728 catalog_manager.cc:1259] Loaded cluster ID: d7701a0e7e41458b999c32ddf734fe23
I20260618 23:30:39.420327 30728 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 48dd0cb815984105abe9300422576a56: loading cluster ID for follower catalog manager: success
I20260618 23:30:39.421205 30728 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 48dd0cb815984105abe9300422576a56: acquiring CA information for follower catalog manager: success
I20260618 23:30:39.421516 30728 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 48dd0cb815984105abe9300422576a56: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260618 23:30:39.457255 30751 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2aca02174f9e4433bfea288882e1e693: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260618 23:30:39.539886 30825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:39.537438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53380 (local address 127.8.39.189:38543)
0618 23:30:39.537565 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:39.537567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:39.537576 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:30:39.537602 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:39.537603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:39.537651 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:39.537724 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:30:39.537910 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:39.538393 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:39.539115 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:39.539296 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:39.539345 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:39.539475 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:30:39.539580 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:30:39.539640 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:39.539746 (+ 106us) server_negotiation.cc:299] Negotiation successful
0618 23:30:39.539765 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":78,"threads_started":1}
I20260618 23:30:39.544059 30831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:39.541748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49718 (local address 127.8.39.188:42657)
0618 23:30:39.541869 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:39.541871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:39.541880 (+ 9us) server_negotiation.cc:407] Connection header received
0618 23:30:39.541917 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:39.541917 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:39.541951 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:39.542023 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:30:39.542116 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:39.542593 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:39.543329 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:39.543529 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:39.543587 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:39.543714 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:30:39.543824 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:30:39.543883 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:39.543918 (+ 35us) server_negotiation.cc:299] Negotiation successful
0618 23:30:39.543935 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260618 23:30:39.549374 30831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:39.546215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49730 (local address 127.8.39.188:42657)
0618 23:30:39.546258 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:39.546261 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:39.546546 (+ 285us) server_negotiation.cc:407] Connection header received
0618 23:30:39.546653 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:39.546654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:39.546665 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:39.546710 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:30:39.546876 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:39.547563 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:39.548479 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:39.548763 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:39.548868 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:39.549039 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:30:39.549079 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:30:39.549171 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:39.549264 (+ 93us) server_negotiation.cc:299] Negotiation successful
0618 23:30:39.549301 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260618 23:30:39.549379 30825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:39.546269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53384 (local address 127.8.39.189:38543)
0618 23:30:39.546342 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:39.546364 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:39.546437 (+ 73us) server_negotiation.cc:407] Connection header received
0618 23:30:39.546472 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:39.546473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:39.546485 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:39.546547 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:30:39.546688 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:39.547458 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:39.548579 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:39.548873 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:39.548933 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:39.549088 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:30:39.549146 (+ 58us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:30:39.549177 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:39.549295 (+ 118us) server_negotiation.cc:299] Negotiation successful
0618 23:30:39.549318 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260618 23:30:39.550552 30838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0618 23:30:39.546177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38866 (local address 127.8.39.190:44193)
0618 23:30:39.546282 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0618 23:30:39.546284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0618 23:30:39.546400 (+ 116us) server_negotiation.cc:407] Connection header received
0618 23:30:39.546489 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0618 23:30:39.546490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0618 23:30:39.546540 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0618 23:30:39.546620 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0618 23:30:39.546749 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:39.547288 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:39.548500 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0618 23:30:39.548780 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0618 23:30:39.548860 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0618 23:30:39.549033 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0618 23:30:39.549163 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0618 23:30:39.549225 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0618 23:30:39.550016 (+ 791us) server_negotiation.cc:299] Negotiation successful
0618 23:30:39.550492 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":54,"threads_started":1}
I20260618 23:30:39.553211 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 30471
I20260618 23:30:39.558799 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 30546
I20260618 23:30:39.564513 8350 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskNmhGNv/build/release/bin/kudu with pid 30621
2026-06-18T23:30:39Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2874 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5763 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (386030 ms total)
[ PASSED ] 54 tests.
[ FAILED ] 2 tests, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
[ FAILED ] SecurityITest.TestEncryptionWithKMSIntegration
2 FAILED TESTS